Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X...X......X] duration: 1m0.003127645s trace: https://jaeger.dev.storj.io/trace/46b48df111d99581 FAIL (2025-12-30 06:14:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 402256s) [noise=true , address=77.90.31.17:4794 , piece_id=5FI7ZF2GFCHPDKEB7F3U5NMXZFGAJEDUFVZA5H4U5HJJI2OTB5EQ, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.896086983 (590h44m2.767170058s) [noise=true , address=77.90.31.19:4857 , piece_id=WGGMF7ZW4HJROMZ3HT7PHY3LWSFFGQSZNDXZYFMIY4LIA7736FSA, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.896896775 (590h44m2.76797982s) [noise=true , address=145.79.227.34:4919 , piece_id=K5HTPRIPPAJBRFLGJMTXMUBQYP2VUKSSFBIDZJ42YXKGZ6NRHP7Q, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.897704834 (590h44m2.768787869s) [noise=true , address=64.49.13.169:4709 , piece_id=4QGR676MN3UCO7VOCOLCSMY2EEVDN4E2BQEZ6T73BBBUFEY7F7CQ, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.898734705 (590h44m2.76981775s) [noise=true , address=77.90.31.11:4580 , piece_id=X5XOX26FSYNEM6E5Y2FHBVSOBDVU5XJ6Z7XFRSPQSHIOPIM4ILLQ, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.900012478 (590h44m2.771095523s) [noise=true , address=193.149.164.143:4699 , piece_id=MVZB6VSTMP3J44LDQDK7QXIA27AB3XLPIGYNHLZDXJODYOSTN3EQ, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.902485407 (590h44m2.773568462s) [noise=true , address=64.49.13.169:4716 , piece_id=3HZZKO6XDXO2JQCUOIXIWUKWRQ3WESRYW3TPRQWTFTM6UYUE4VOQ, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.902823431 (590h44m2.773906476s) [noise=true , address=77.90.31.14:4688 , piece_id=XJU7EHH2573FOVLOSLGAA46JEZG2AIKLVKZOXRBYYM7X56E22RGA, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.903164710 (590h44m2.774247755s) [noise=true , address=45.91.116.11:4139 , piece_id=CKEVNQMM5MAMUE4XHWYN6SD4MWZB5CMMYAB7JDZ3C43NNMDSJVCA, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.903544323 (590h44m2.774627358s) [noise=true , address=145.79.227.30:4771 , piece_id=U3PVHRB7KCA76GXAO3SJJOED3NUEJ3B3TXXQE6SX4DJEM6M5O4RA, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.905176611 (590h44m2.776259656s) [noise=true , address=145.79.227.11:4735 , piece_id=ENOSVX5NS3JUUCILOYYKHDZHOUSJZD6THCW365EYZLOY3BSE6D5A, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.906736873 (590h44m2.777819908s) [noise=true , address=77.90.31.15:4717 , piece_id=4X6UXT3HTTALGUBA4VAIMQS6P6P5JLY7FVDAFWL2PK2DUVZDBPZA, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.907198291 (590h44m2.778281336s) [noise=true , address=77.90.31.18:4833 , piece_id=5OHYRK74NRJAS6ENJIEEP3NXETW2GIOHESUCHHLIUCOO5E3TENPQ, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.908486875 (590h44m2.77956991s) [noise=true , address=145.79.227.33:4898 , piece_id=4J7VGL3KNWH3GZ4ON4KXZLLCZLBUGHROKIIB7WGY63RHH5IPF2IA, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:02.908969063 (590h44m2.780052108s) [noise=true , address=145.79.227.13:4832 , piece_id=JVI5TVAKEYPCIQJ4465JGHH4DO6UR6VJTAAIECE7EWVECNBNCIPA, seg_pos={0 0}, upload=517864]: Uploading piece... Dec 30 06:13:04.360994985 (590h44m4.23207803s) [upload=517864]: Next returned nil segment Dec 30 06:13:04.361008010 (590h44m4.232091045s) [upload=517864]: Waiting for error group managing segments... Dec 30 06:13:04.781371641 (590h44m4.652454686s) [noise=true , address=145.79.227.34:4919 , piece_id=K5HTPRIPPAJBRFLGJMTXMUBQYP2VUKSSFBIDZJ42YXKGZ6NRHP7Q, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:04.784208242 (590h44m4.655291288s) [noise=true , address=145.79.227.11:4735 , piece_id=ENOSVX5NS3JUUCILOYYKHDZHOUSJZD6THCW365EYZLOY3BSE6D5A, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:04.836043685 (590h44m4.70712673s) [noise=true , address=145.79.227.35:4486 , piece_id=KW3PACCWHZXZPQ5TKPSRSYFSHBXN5XMVH3XXOVEZUR3SF3JFOUVA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:04.866039612 (590h44m4.737122657s) [noise=true , address=145.79.227.13:4832 , piece_id=JVI5TVAKEYPCIQJ4465JGHH4DO6UR6VJTAAIECE7EWVECNBNCIPA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:05.063568059 (590h44m4.934651124s) [noise=true , address=77.90.31.11:4580 , piece_id=X5XOX26FSYNEM6E5Y2FHBVSOBDVU5XJ6Z7XFRSPQSHIOPIM4ILLQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:05.071636672 (590h44m4.942719727s) [noise=true , address=77.90.31.19:4857 , piece_id=WGGMF7ZW4HJROMZ3HT7PHY3LWSFFGQSZNDXZYFMIY4LIA7736FSA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:05.088316663 (590h44m4.959399708s) [noise=true , address=77.90.31.17:4794 , piece_id=5FI7ZF2GFCHPDKEB7F3U5NMXZFGAJEDUFVZA5H4U5HJJI2OTB5EQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:05.170834401 (590h44m5.041917446s) [noise=true , address=77.90.31.15:4717 , piece_id=4X6UXT3HTTALGUBA4VAIMQS6P6P5JLY7FVDAFWL2PK2DUVZDBPZA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:05.272783788 (590h44m5.143866834s) [noise=true , address=77.90.31.14:4688 , piece_id=XJU7EHH2573FOVLOSLGAA46JEZG2AIKLVKZOXRBYYM7X56E22RGA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:05.667537083 (590h44m5.538620128s) [noise=true , address=145.79.227.43:4189 , piece_id=FZK7EWS2AKZZZL4B45TAR3SVPJ2UYFJGR35HIIPU3FUEWHRSX36A, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:05.691236510 (590h44m5.562319555s) [noise=true , address=193.149.164.143:4699 , piece_id=MVZB6VSTMP3J44LDQDK7QXIA27AB3XLPIGYNHLZDXJODYOSTN3EQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:05.712297864 (590h44m5.583380899s) [noise=true , address=145.79.227.30:4771 , piece_id=U3PVHRB7KCA76GXAO3SJJOED3NUEJ3B3TXXQE6SX4DJEM6M5O4RA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:05.720879355 (590h44m5.591962401s) [noise=true , address=145.79.227.40:4082 , piece_id=4BS3GCGUFDSZLFXPPXAPT2524TJRLZT6GMH6Y76HLIFQDHXNPKBA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:05.954725096 (590h44m5.825808141s) [noise=true , address=145.79.227.33:4898 , piece_id=4J7VGL3KNWH3GZ4ON4KXZLLCZLBUGHROKIIB7WGY63RHH5IPF2IA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil> Dec 30 06:13:05.954752869 (590h44m5.825835914s) [seg_pos={0 0}, upload=517864]: Segment reached optimal threshold of 14 pieces. Dec 30 06:13:05.955016099 (590h44m5.826099144s) [noise=true , address=64.49.13.169:4709 , piece_id=4QGR676MN3UCO7VOCOLCSMY2EEVDN4E2BQEZ6T73BBBUFEY7F7CQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:49652->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:49652->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49652->64.49.13.169:4709: use of closed network connection Dec 30 06:13:05.955027190 (590h44m5.826110225s) [noise=true , address=64.49.13.169:4709 , piece_id=4QGR676MN3UCO7VOCOLCSMY2EEVDN4E2BQEZ6T73BBBUFEY7F7CQ, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold Dec 30 06:13:05.955054612 (590h44m5.826137658s) [noise=true , address=45.91.116.11:4139 , piece_id=CKEVNQMM5MAMUE4XHWYN6SD4MWZB5CMMYAB7JDZ3C43NNMDSJVCA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD): protocol: expected piece hash; context canceled; EOF Dec 30 06:13:05.955062788 (590h44m5.826145833s) [noise=true , address=45.91.116.11:4139 , piece_id=CKEVNQMM5MAMUE4XHWYN6SD4MWZB5CMMYAB7JDZ3C43NNMDSJVCA, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold Dec 30 06:13:05.955166545 (590h44m5.82624959s) [noise=true , address=77.90.31.18:4833 , piece_id=5OHYRK74NRJAS6ENJIEEP3NXETW2GIOHESUCHHLIUCOO5E3TENPQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 30 06:13:05.955172297 (590h44m5.826255342s) [noise=true , address=77.90.31.18:4833 , piece_id=5OHYRK74NRJAS6ENJIEEP3NXETW2GIOHESUCHHLIUCOO5E3TENPQ, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold Dec 30 06:13:05.955273479 (590h44m5.826356524s) [noise=true , address=64.49.13.169:4716 , piece_id=3HZZKO6XDXO2JQCUOIXIWUKWRQ3WESRYW3TPRQWTFTM6UYUE4VOQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:49808->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:49808->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49808->64.49.13.169:4716: use of closed network connection Dec 30 06:13:05.955280313 (590h44m5.826363348s) [noise=true , address=64.49.13.169:4716 , piece_id=3HZZKO6XDXO2JQCUOIXIWUKWRQ3WESRYW3TPRQWTFTM6UYUE4VOQ, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold Dec 30 06:13:05.955310550 (590h44m5.826393596s) [seg_pos={0 0}, upload=517864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:13:05.955381786 (590h44m5.826464821s) [upload=517864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:13:05.955386705 (590h44m5.82646974s) [upload=517864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:13:06.243506482 (590h44m6.114589527s) [upload=517864]: Done uploading segments... Dec 30 06:13:06.243537802 (590h44m6.114620837s) [upload=517864]: Upload finished. err: <nil> Duration: 1m0.003144518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X.......X..X..X....X.XX.X..X..X..X..XX.......X..X......X..] duration: 6.12020922s trace: https://jaeger.dev.storj.io/trace/5fefb10a11410f FAIL (2025-12-30 06:13:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: HZLSG45ZFSD4ZCALWM6H3UQGQQ6B42ZBAWNTTYFY3LCQ6GYPGWRQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 2UGZEMFX2KKCAA4TORR3IIZNZPCQ7M6GOXGX64SV4ULYRKX5IZNQ) context deadline exceeded ======LOGS====== Dec 30 06:12:49.572115696 (590h43m49.443198731s) [upload=517860]: Starting upload Dec 30 06:12:49.572177864 (590h43m49.443260899s) [upload=517860]: Done starting upload Dec 30 06:12:49.572294476 (590h43m49.443377511s) [upload=517860]: Uploading segments... Dec 30 06:12:49.572389808 (590h43m49.443472843s) [upload=517860]: Got next segment. Inline: false Dec 30 06:12:49.572423242 (590h43m49.443506287s) [upload=517860]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:12:49.572428532 (590h43m49.443511567s) [upload=517860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:12:49.823225820 (590h43m49.694308855s) [seg_pos={0 0}, upload=517860]: Begin upload segment... Dec 30 06:12:49.823280284 (590h43m49.694363329s) [seg_pos={0 0}, upload=517860]: Done begin upload segment. Dec 30 06:12:49.823322594 (590h43m49.694405629s) [noise=true , address=145.79.227.14:4838 , piece_id=DAH6ZELJGNIBEGFP5FXYRO2VNJ236ZGHGK6DAJNA3X5ZPLTB7F3Q, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.823425169 (590h43m49.694508224s) [noise=true , address=145.79.227.31:4799 , piece_id=IH5CCLJ4ZPHKI4D2VNJJGFKXXITSVLWQ4FXAK5YIIKY2IKUXRTRQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.824011586 (590h43m49.695094631s) [noise=true , address=77.90.31.12:4589 , piece_id=T5PFZHHONH6ZU336I2HQZGD2E7OHR5RVH3MDLASSRJUYCLJTD4EQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.824450132 (590h43m49.695533177s) [noise=true , address=77.90.31.18:4805 , piece_id=HZLSG45ZFSD4ZCALWM6H3UQGQQ6B42ZBAWNTTYFY3LCQ6GYPGWRQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.824771714 (590h43m49.695854749s) [noise=true , address=145.79.227.5:4221 , piece_id=OWGDLM64DW2VX4JBOL3BAFFOFQSYFSXUTCZ5ZFQEKPDUVUA2UTWA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.825067597 (590h43m49.696150632s) [noise=true , address=145.79.227.37:4566 , piece_id=XSGG3PWLTLSQOJSOYCU6APWZM4OQKYI3QR4V4TTYEOB62FXQF3OA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.825360816 (590h43m49.696443861s) [noise=true , address=77.90.31.21:4877 , piece_id=2UGZEMFX2KKCAA4TORR3IIZNZPCQ7M6GOXGX64SV4ULYRKX5IZNQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.825644065 (590h43m49.6967271s) [noise=true , address=77.90.31.19:4844 , piece_id=JHBLO56E2YBUTGIY7XZJRG3BYWDY6QEFKZNYYV6CNOL7FUN62GSQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.826064385 (590h43m49.69714742s) [noise=true , address=145.79.227.38:4977 , piece_id=SV7GECWG5XL7YOH2WTBCXFRWOB7LYZMM3454XJJEMV3STY7SOI2Q, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.827417372 (590h43m49.698500407s) [noise=true , address=145.79.227.33:4871 , piece_id=7MAICINKRVUOR2EEAHBMTXRNGKHBXX3HO37THFRETEBX7FIYKSJQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.828234747 (590h43m49.699317783s) [noise=true , address=145.79.227.50:4375 , piece_id=DFV6SJOWTLYXYZFJJFKAHNE3GTT2UN5OCFRM4T6LAE6JFCOAUSWQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.829708655 (590h43m49.7007917s) [noise=true , address=145.79.227.13:4802 , piece_id=2UROW7VPQMNW2RKBRLXMFW5WJNFLLTHRMYT7MPPNPFY4FVV35BDA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.831118058 (590h43m49.702201103s) [noise=true , address=145.79.227.3:4150 , piece_id=6OLRN57ONNQRQ2PL73TR3P4USTYXZSRKWOIFGCJ6ERMHCKQFEB6A, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.832323093 (590h43m49.703406138s) [noise=true , address=45.91.116.14:4192 , piece_id=PWIOIBHAOFQVWQ3TXMTYMAZ7JSMNS4IO4TEXDW3HSVYIHOZH32UA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.833524049 (590h43m49.704607084s) [noise=true , address=145.79.227.39:4013 , piece_id=U44AWCUTDBZLR6KDOYORSPEAPZHOQGP6J7IQJTBJ6DG4RFLXSOBQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.834792686 (590h43m49.705875731s) [noise=true , address=145.79.227.12:4767 , piece_id=4SALTYL6TE6PF3CYBJ7RLB7GINPRLJ5W4ONKDSVNJJ7PSRXIZBMA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:51.537747536 (590h43m51.408830571s) [upload=517860]: Next returned nil segment Dec 30 06:12:51.537764830 (590h43m51.408847865s) [upload=517860]: Waiting for error group managing segments... Dec 30 06:12:51.824065021 (590h43m51.695148056s) [noise=true , address=145.79.227.13:4802 , piece_id=2UROW7VPQMNW2RKBRLXMFW5WJNFLLTHRMYT7MPPNPFY4FVV35BDA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.827230819 (590h43m51.698313865s) [noise=true , address=145.79.227.12:4767 , piece_id=4SALTYL6TE6PF3CYBJ7RLB7GINPRLJ5W4ONKDSVNJJ7PSRXIZBMA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.827858805 (590h43m51.69894185s) [noise=true , address=145.79.227.31:4799 , piece_id=IH5CCLJ4ZPHKI4D2VNJJGFKXXITSVLWQ4FXAK5YIIKY2IKUXRTRQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.832703911 (590h43m51.703786946s) [noise=true , address=145.79.227.50:4375 , piece_id=DFV6SJOWTLYXYZFJJFKAHNE3GTT2UN5OCFRM4T6LAE6JFCOAUSWQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.835291237 (590h43m51.706374282s) [noise=true , address=145.79.227.38:4977 , piece_id=SV7GECWG5XL7YOH2WTBCXFRWOB7LYZMM3454XJJEMV3STY7SOI2Q, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.836848182 (590h43m51.707931227s) [noise=true , address=145.79.227.14:4838 , piece_id=DAH6ZELJGNIBEGFP5FXYRO2VNJ236ZGHGK6DAJNA3X5ZPLTB7F3Q, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.837531233 (590h43m51.708614268s) [noise=true , address=145.79.227.37:4566 , piece_id=XSGG3PWLTLSQOJSOYCU6APWZM4OQKYI3QR4V4TTYEOB62FXQF3OA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.848758241 (590h43m51.719841286s) [noise=true , address=145.79.227.3:4150 , piece_id=6OLRN57ONNQRQ2PL73TR3P4USTYXZSRKWOIFGCJ6ERMHCKQFEB6A, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.850321468 (590h43m51.721404503s) [noise=true , address=145.79.227.39:4013 , piece_id=U44AWCUTDBZLR6KDOYORSPEAPZHOQGP6J7IQJTBJ6DG4RFLXSOBQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.850758630 (590h43m51.721841665s) [noise=true , address=145.79.227.5:4221 , piece_id=OWGDLM64DW2VX4JBOL3BAFFOFQSYFSXUTCZ5ZFQEKPDUVUA2UTWA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.080630302 (590h43m51.951713348s) [noise=true , address=77.90.31.18:4805 , piece_id=HZLSG45ZFSD4ZCALWM6H3UQGQQ6B42ZBAWNTTYFY3LCQ6GYPGWRQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.100633332 (590h43m51.971716377s) [noise=true , address=77.90.31.21:4877 , piece_id=2UGZEMFX2KKCAA4TORR3IIZNZPCQ7M6GOXGX64SV4ULYRKX5IZNQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.121440091 (590h43m51.992523126s) [noise=true , address=77.90.31.19:4844 , piece_id=JHBLO56E2YBUTGIY7XZJRG3BYWDY6QEFKZNYYV6CNOL7FUN62GSQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.121491529 (590h43m51.992574565s) [seg_pos={0 0}, upload=517860]: Segment reached optimal threshold of 13 pieces. Dec 30 06:12:52.121685108 (590h43m51.992768143s) [noise=true , address=45.91.116.14:4192 , piece_id=PWIOIBHAOFQVWQ3TXMTYMAZ7JSMNS4IO4TEXDW3HSVYIHOZH32UA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 06:12:52.121693774 (590h43m51.992776809s) [noise=true , address=45.91.116.14:4192 , piece_id=PWIOIBHAOFQVWQ3TXMTYMAZ7JSMNS4IO4TEXDW3HSVYIHOZH32UA, seg_pos={0 0}, upload=517860]: Long tail cancellation due to optimal threshold Dec 30 06:12:52.121903162 (590h43m51.992986238s) [noise=true , address=145.79.227.33:4871 , piece_id=7MAICINKRVUOR2EEAHBMTXRNGKHBXX3HO37THFRETEBX7FIYKSJQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 06:12:52.121923882 (590h43m51.993006917s) [noise=true , address=145.79.227.33:4871 , piece_id=7MAICINKRVUOR2EEAHBMTXRNGKHBXX3HO37THFRETEBX7FIYKSJQ, seg_pos={0 0}, upload=517860]: Long tail cancellation due to optimal threshold Dec 30 06:12:52.121942137 (590h43m51.993025172s) [noise=true , address=77.90.31.12:4589 , piece_id=T5PFZHHONH6ZU336I2HQZGD2E7OHR5RVH3MDLASSRJUYCLJTD4EQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 30 06:12:52.121949812 (590h43m51.993032857s) [noise=true , address=77.90.31.12:4589 , piece_id=T5PFZHHONH6ZU336I2HQZGD2E7OHR5RVH3MDLASSRJUYCLJTD4EQ, seg_pos={0 0}, upload=517860]: Long tail cancellation due to optimal threshold Dec 30 06:12:52.121967816 (590h43m51.993050851s) [seg_pos={0 0}, upload=517860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:12:52.122056395 (590h43m51.99313943s) [upload=517860]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:12:52.122064029 (590h43m51.993147064s) [upload=517860]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:12:52.412694056 (590h43m52.283777111s) [upload=517860]: Done uploading segments... Dec 30 06:12:52.412721609 (590h43m52.283804654s) [upload=517860]: Upload finished. err: <nil> Duration: 6.120233897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X.......X..X..X....X.XX.X..X..X..X..XX.......X..X......X.] duration: 5.909492758s trace: https://jaeger.dev.storj.io/trace/5fefb10a11410f FAIL (2025-12-30 06:13:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: HZLSG45ZFSD4ZCALWM6H3UQGQQ6B42ZBAWNTTYFY3LCQ6GYPGWRQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 2UGZEMFX2KKCAA4TORR3IIZNZPCQ7M6GOXGX64SV4ULYRKX5IZNQ) context deadline exceeded ======LOGS====== Dec 30 06:12:49.572115696 (590h43m49.443198731s) [upload=517860]: Starting upload Dec 30 06:12:49.572177864 (590h43m49.443260899s) [upload=517860]: Done starting upload Dec 30 06:12:49.572294476 (590h43m49.443377511s) [upload=517860]: Uploading segments... Dec 30 06:12:49.572389808 (590h43m49.443472843s) [upload=517860]: Got next segment. Inline: false Dec 30 06:12:49.572423242 (590h43m49.443506287s) [upload=517860]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:12:49.572428532 (590h43m49.443511567s) [upload=517860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:12:49.823225820 (590h43m49.694308855s) [seg_pos={0 0}, upload=517860]: Begin upload segment... Dec 30 06:12:49.823280284 (590h43m49.694363329s) [seg_pos={0 0}, upload=517860]: Done begin upload segment. Dec 30 06:12:49.823322594 (590h43m49.694405629s) [noise=true , address=145.79.227.14:4838 , piece_id=DAH6ZELJGNIBEGFP5FXYRO2VNJ236ZGHGK6DAJNA3X5ZPLTB7F3Q, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.823425169 (590h43m49.694508224s) [noise=true , address=145.79.227.31:4799 , piece_id=IH5CCLJ4ZPHKI4D2VNJJGFKXXITSVLWQ4FXAK5YIIKY2IKUXRTRQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.824011586 (590h43m49.695094631s) [noise=true , address=77.90.31.12:4589 , piece_id=T5PFZHHONH6ZU336I2HQZGD2E7OHR5RVH3MDLASSRJUYCLJTD4EQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.824450132 (590h43m49.695533177s) [noise=true , address=77.90.31.18:4805 , piece_id=HZLSG45ZFSD4ZCALWM6H3UQGQQ6B42ZBAWNTTYFY3LCQ6GYPGWRQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.824771714 (590h43m49.695854749s) [noise=true , address=145.79.227.5:4221 , piece_id=OWGDLM64DW2VX4JBOL3BAFFOFQSYFSXUTCZ5ZFQEKPDUVUA2UTWA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.825067597 (590h43m49.696150632s) [noise=true , address=145.79.227.37:4566 , piece_id=XSGG3PWLTLSQOJSOYCU6APWZM4OQKYI3QR4V4TTYEOB62FXQF3OA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.825360816 (590h43m49.696443861s) [noise=true , address=77.90.31.21:4877 , piece_id=2UGZEMFX2KKCAA4TORR3IIZNZPCQ7M6GOXGX64SV4ULYRKX5IZNQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.825644065 (590h43m49.6967271s) [noise=true , address=77.90.31.19:4844 , piece_id=JHBLO56E2YBUTGIY7XZJRG3BYWDY6QEFKZNYYV6CNOL7FUN62GSQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.826064385 (590h43m49.69714742s) [noise=true , address=145.79.227.38:4977 , piece_id=SV7GECWG5XL7YOH2WTBCXFRWOB7LYZMM3454XJJEMV3STY7SOI2Q, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.827417372 (590h43m49.698500407s) [noise=true , address=145.79.227.33:4871 , piece_id=7MAICINKRVUOR2EEAHBMTXRNGKHBXX3HO37THFRETEBX7FIYKSJQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.828234747 (590h43m49.699317783s) [noise=true , address=145.79.227.50:4375 , piece_id=DFV6SJOWTLYXYZFJJFKAHNE3GTT2UN5OCFRM4T6LAE6JFCOAUSWQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.829708655 (590h43m49.7007917s) [noise=true , address=145.79.227.13:4802 , piece_id=2UROW7VPQMNW2RKBRLXMFW5WJNFLLTHRMYT7MPPNPFY4FVV35BDA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.831118058 (590h43m49.702201103s) [noise=true , address=145.79.227.3:4150 , piece_id=6OLRN57ONNQRQ2PL73TR3P4USTYXZSRKWOIFGCJ6ERMHCKQFEB6A, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.832323093 (590h43m49.703406138s) [noise=true , address=45.91.116.14:4192 , piece_id=PWIOIBHAOFQVWQ3TXMTYMAZ7JSMNS4IO4TEXDW3HSVYIHOZH32UA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.833524049 (590h43m49.704607084s) [noise=true , address=145.79.227.39:4013 , piece_id=U44AWCUTDBZLR6KDOYORSPEAPZHOQGP6J7IQJTBJ6DG4RFLXSOBQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.834792686 (590h43m49.705875731s) [noise=true , address=145.79.227.12:4767 , piece_id=4SALTYL6TE6PF3CYBJ7RLB7GINPRLJ5W4ONKDSVNJJ7PSRXIZBMA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:51.537747536 (590h43m51.408830571s) [upload=517860]: Next returned nil segment Dec 30 06:12:51.537764830 (590h43m51.408847865s) [upload=517860]: Waiting for error group managing segments... Dec 30 06:12:51.824065021 (590h43m51.695148056s) [noise=true , address=145.79.227.13:4802 , piece_id=2UROW7VPQMNW2RKBRLXMFW5WJNFLLTHRMYT7MPPNPFY4FVV35BDA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.827230819 (590h43m51.698313865s) [noise=true , address=145.79.227.12:4767 , piece_id=4SALTYL6TE6PF3CYBJ7RLB7GINPRLJ5W4ONKDSVNJJ7PSRXIZBMA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.827858805 (590h43m51.69894185s) [noise=true , address=145.79.227.31:4799 , piece_id=IH5CCLJ4ZPHKI4D2VNJJGFKXXITSVLWQ4FXAK5YIIKY2IKUXRTRQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.832703911 (590h43m51.703786946s) [noise=true , address=145.79.227.50:4375 , piece_id=DFV6SJOWTLYXYZFJJFKAHNE3GTT2UN5OCFRM4T6LAE6JFCOAUSWQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.835291237 (590h43m51.706374282s) [noise=true , address=145.79.227.38:4977 , piece_id=SV7GECWG5XL7YOH2WTBCXFRWOB7LYZMM3454XJJEMV3STY7SOI2Q, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.836848182 (590h43m51.707931227s) [noise=true , address=145.79.227.14:4838 , piece_id=DAH6ZELJGNIBEGFP5FXYRO2VNJ236ZGHGK6DAJNA3X5ZPLTB7F3Q, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.837531233 (590h43m51.708614268s) [noise=true , address=145.79.227.37:4566 , piece_id=XSGG3PWLTLSQOJSOYCU6APWZM4OQKYI3QR4V4TTYEOB62FXQF3OA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.848758241 (590h43m51.719841286s) [noise=true , address=145.79.227.3:4150 , piece_id=6OLRN57ONNQRQ2PL73TR3P4USTYXZSRKWOIFGCJ6ERMHCKQFEB6A, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.850321468 (590h43m51.721404503s) [noise=true , address=145.79.227.39:4013 , piece_id=U44AWCUTDBZLR6KDOYORSPEAPZHOQGP6J7IQJTBJ6DG4RFLXSOBQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.850758630 (590h43m51.721841665s) [noise=true , address=145.79.227.5:4221 , piece_id=OWGDLM64DW2VX4JBOL3BAFFOFQSYFSXUTCZ5ZFQEKPDUVUA2UTWA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.080630302 (590h43m51.951713348s) [noise=true , address=77.90.31.18:4805 , piece_id=HZLSG45ZFSD4ZCALWM6H3UQGQQ6B42ZBAWNTTYFY3LCQ6GYPGWRQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.100633332 (590h43m51.971716377s) [noise=true , address=77.90.31.21:4877 , piece_id=2UGZEMFX2KKCAA4TORR3IIZNZPCQ7M6GOXGX64SV4ULYRKX5IZNQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.121440091 (590h43m51.992523126s) [noise=true , address=77.90.31.19:4844 , piece_id=JHBLO56E2YBUTGIY7XZJRG3BYWDY6QEFKZNYYV6CNOL7FUN62GSQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.121491529 (590h43m51.992574565s) [seg_pos={0 0}, upload=517860]: Segment reached optimal threshold of 13 pieces. Dec 30 06:12:52.121685108 (590h43m51.992768143s) [noise=true , address=45.91.116.14:4192 , piece_id=PWIOIBHAOFQVWQ3TXMTYMAZ7JSMNS4IO4TEXDW3HSVYIHOZH32UA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 06:12:52.121693774 (590h43m51.992776809s) [noise=true , address=45.91.116.14:4192 , piece_id=PWIOIBHAOFQVWQ3TXMTYMAZ7JSMNS4IO4TEXDW3HSVYIHOZH32UA, seg_pos={0 0}, upload=517860]: Long tail cancellation due to optimal threshold Dec 30 06:12:52.121903162 (590h43m51.992986238s) [noise=true , address=145.79.227.33:4871 , piece_id=7MAICINKRVUOR2EEAHBMTXRNGKHBXX3HO37THFRETEBX7FIYKSJQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 06:12:52.121923882 (590h43m51.993006917s) [noise=true , address=145.79.227.33:4871 , piece_id=7MAICINKRVUOR2EEAHBMTXRNGKHBXX3HO37THFRETEBX7FIYKSJQ, seg_pos={0 0}, upload=517860]: Long tail cancellation due to optimal threshold Dec 30 06:12:52.121942137 (590h43m51.993025172s) [noise=true , address=77.90.31.12:4589 , piece_id=T5PFZHHONH6ZU336I2HQZGD2E7OHR5RVH3MDLASSRJUYCLJTD4EQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 30 06:12:52.121949812 (590h43m51.993032857s) [noise=true , address=77.90.31.12:4589 , piece_id=T5PFZHHONH6ZU336I2HQZGD2E7OHR5RVH3MDLASSRJUYCLJTD4EQ, seg_pos={0 0}, upload=517860]: Long tail cancellation due to optimal threshold Dec 30 06:12:52.121967816 (590h43m51.993050851s) [seg_pos={0 0}, upload=517860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:12:52.122056395 (590h43m51.99313943s) [upload=517860]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:12:52.122064029 (590h43m51.993147064s) [upload=517860]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:12:52.412694056 (590h43m52.283777111s) [upload=517860]: Done uploading segments... Dec 30 06:12:52.412721609 (590h43m52.283804654s) [upload=517860]: Upload finished. err: <nil> Duration: 5.909523075s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.X.......X..X..X....X.XX.X..X..X..X..XX.......X..X......X] duration: 30.002860234s trace: https://jaeger.dev.storj.io/trace/5fefb10a11410f FAIL (2025-12-30 06:13:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: HZLSG45ZFSD4ZCALWM6H3UQGQQ6B42ZBAWNTTYFY3LCQ6GYPGWRQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 2UGZEMFX2KKCAA4TORR3IIZNZPCQ7M6GOXGX64SV4ULYRKX5IZNQ) context deadline exceeded ======LOGS====== Dec 30 06:12:49.572115696 (590h43m49.443198731s) [upload=517860]: Starting upload Dec 30 06:12:49.572177864 (590h43m49.443260899s) [upload=517860]: Done starting upload Dec 30 06:12:49.572294476 (590h43m49.443377511s) [upload=517860]: Uploading segments... Dec 30 06:12:49.572389808 (590h43m49.443472843s) [upload=517860]: Got next segment. Inline: false Dec 30 06:12:49.572423242 (590h43m49.443506287s) [upload=517860]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:12:49.572428532 (590h43m49.443511567s) [upload=517860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:12:49.823225820 (590h43m49.694308855s) [seg_pos={0 0}, upload=517860]: Begin upload segment... Dec 30 06:12:49.823280284 (590h43m49.694363329s) [seg_pos={0 0}, upload=517860]: Done begin upload segment. Dec 30 06:12:49.823322594 (590h43m49.694405629s) [noise=true , address=145.79.227.14:4838 , piece_id=DAH6ZELJGNIBEGFP5FXYRO2VNJ236ZGHGK6DAJNA3X5ZPLTB7F3Q, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.823425169 (590h43m49.694508224s) [noise=true , address=145.79.227.31:4799 , piece_id=IH5CCLJ4ZPHKI4D2VNJJGFKXXITSVLWQ4FXAK5YIIKY2IKUXRTRQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.824011586 (590h43m49.695094631s) [noise=true , address=77.90.31.12:4589 , piece_id=T5PFZHHONH6ZU336I2HQZGD2E7OHR5RVH3MDLASSRJUYCLJTD4EQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.824450132 (590h43m49.695533177s) [noise=true , address=77.90.31.18:4805 , piece_id=HZLSG45ZFSD4ZCALWM6H3UQGQQ6B42ZBAWNTTYFY3LCQ6GYPGWRQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.824771714 (590h43m49.695854749s) [noise=true , address=145.79.227.5:4221 , piece_id=OWGDLM64DW2VX4JBOL3BAFFOFQSYFSXUTCZ5ZFQEKPDUVUA2UTWA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.825067597 (590h43m49.696150632s) [noise=true , address=145.79.227.37:4566 , piece_id=XSGG3PWLTLSQOJSOYCU6APWZM4OQKYI3QR4V4TTYEOB62FXQF3OA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.825360816 (590h43m49.696443861s) [noise=true , address=77.90.31.21:4877 , piece_id=2UGZEMFX2KKCAA4TORR3IIZNZPCQ7M6GOXGX64SV4ULYRKX5IZNQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.825644065 (590h43m49.6967271s) [noise=true , address=77.90.31.19:4844 , piece_id=JHBLO56E2YBUTGIY7XZJRG3BYWDY6QEFKZNYYV6CNOL7FUN62GSQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.826064385 (590h43m49.69714742s) [noise=true , address=145.79.227.38:4977 , piece_id=SV7GECWG5XL7YOH2WTBCXFRWOB7LYZMM3454XJJEMV3STY7SOI2Q, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.827417372 (590h43m49.698500407s) [noise=true , address=145.79.227.33:4871 , piece_id=7MAICINKRVUOR2EEAHBMTXRNGKHBXX3HO37THFRETEBX7FIYKSJQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.828234747 (590h43m49.699317783s) [noise=true , address=145.79.227.50:4375 , piece_id=DFV6SJOWTLYXYZFJJFKAHNE3GTT2UN5OCFRM4T6LAE6JFCOAUSWQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.829708655 (590h43m49.7007917s) [noise=true , address=145.79.227.13:4802 , piece_id=2UROW7VPQMNW2RKBRLXMFW5WJNFLLTHRMYT7MPPNPFY4FVV35BDA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.831118058 (590h43m49.702201103s) [noise=true , address=145.79.227.3:4150 , piece_id=6OLRN57ONNQRQ2PL73TR3P4USTYXZSRKWOIFGCJ6ERMHCKQFEB6A, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.832323093 (590h43m49.703406138s) [noise=true , address=45.91.116.14:4192 , piece_id=PWIOIBHAOFQVWQ3TXMTYMAZ7JSMNS4IO4TEXDW3HSVYIHOZH32UA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.833524049 (590h43m49.704607084s) [noise=true , address=145.79.227.39:4013 , piece_id=U44AWCUTDBZLR6KDOYORSPEAPZHOQGP6J7IQJTBJ6DG4RFLXSOBQ, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:49.834792686 (590h43m49.705875731s) [noise=true , address=145.79.227.12:4767 , piece_id=4SALTYL6TE6PF3CYBJ7RLB7GINPRLJ5W4ONKDSVNJJ7PSRXIZBMA, seg_pos={0 0}, upload=517860]: Uploading piece... Dec 30 06:12:51.537747536 (590h43m51.408830571s) [upload=517860]: Next returned nil segment Dec 30 06:12:51.537764830 (590h43m51.408847865s) [upload=517860]: Waiting for error group managing segments... Dec 30 06:12:51.824065021 (590h43m51.695148056s) [noise=true , address=145.79.227.13:4802 , piece_id=2UROW7VPQMNW2RKBRLXMFW5WJNFLLTHRMYT7MPPNPFY4FVV35BDA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.827230819 (590h43m51.698313865s) [noise=true , address=145.79.227.12:4767 , piece_id=4SALTYL6TE6PF3CYBJ7RLB7GINPRLJ5W4ONKDSVNJJ7PSRXIZBMA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.827858805 (590h43m51.69894185s) [noise=true , address=145.79.227.31:4799 , piece_id=IH5CCLJ4ZPHKI4D2VNJJGFKXXITSVLWQ4FXAK5YIIKY2IKUXRTRQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.832703911 (590h43m51.703786946s) [noise=true , address=145.79.227.50:4375 , piece_id=DFV6SJOWTLYXYZFJJFKAHNE3GTT2UN5OCFRM4T6LAE6JFCOAUSWQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.835291237 (590h43m51.706374282s) [noise=true , address=145.79.227.38:4977 , piece_id=SV7GECWG5XL7YOH2WTBCXFRWOB7LYZMM3454XJJEMV3STY7SOI2Q, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.836848182 (590h43m51.707931227s) [noise=true , address=145.79.227.14:4838 , piece_id=DAH6ZELJGNIBEGFP5FXYRO2VNJ236ZGHGK6DAJNA3X5ZPLTB7F3Q, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.837531233 (590h43m51.708614268s) [noise=true , address=145.79.227.37:4566 , piece_id=XSGG3PWLTLSQOJSOYCU6APWZM4OQKYI3QR4V4TTYEOB62FXQF3OA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.848758241 (590h43m51.719841286s) [noise=true , address=145.79.227.3:4150 , piece_id=6OLRN57ONNQRQ2PL73TR3P4USTYXZSRKWOIFGCJ6ERMHCKQFEB6A, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.850321468 (590h43m51.721404503s) [noise=true , address=145.79.227.39:4013 , piece_id=U44AWCUTDBZLR6KDOYORSPEAPZHOQGP6J7IQJTBJ6DG4RFLXSOBQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:51.850758630 (590h43m51.721841665s) [noise=true , address=145.79.227.5:4221 , piece_id=OWGDLM64DW2VX4JBOL3BAFFOFQSYFSXUTCZ5ZFQEKPDUVUA2UTWA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.080630302 (590h43m51.951713348s) [noise=true , address=77.90.31.18:4805 , piece_id=HZLSG45ZFSD4ZCALWM6H3UQGQQ6B42ZBAWNTTYFY3LCQ6GYPGWRQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.100633332 (590h43m51.971716377s) [noise=true , address=77.90.31.21:4877 , piece_id=2UGZEMFX2KKCAA4TORR3IIZNZPCQ7M6GOXGX64SV4ULYRKX5IZNQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.121440091 (590h43m51.992523126s) [noise=true , address=77.90.31.19:4844 , piece_id=JHBLO56E2YBUTGIY7XZJRG3BYWDY6QEFKZNYYV6CNOL7FUN62GSQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: <nil> Dec 30 06:12:52.121491529 (590h43m51.992574565s) [seg_pos={0 0}, upload=517860]: Segment reached optimal threshold of 13 pieces. Dec 30 06:12:52.121685108 (590h43m51.992768143s) [noise=true , address=45.91.116.14:4192 , piece_id=PWIOIBHAOFQVWQ3TXMTYMAZ7JSMNS4IO4TEXDW3HSVYIHOZH32UA, seg_pos={0 0}, upload=517860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 06:12:52.121693774 (590h43m51.992776809s) [noise=true , address=45.91.116.14:4192 , piece_id=PWIOIBHAOFQVWQ3TXMTYMAZ7JSMNS4IO4TEXDW3HSVYIHOZH32UA, seg_pos={0 0}, upload=517860]: Long tail cancellation due to optimal threshold Dec 30 06:12:52.121903162 (590h43m51.992986238s) [noise=true , address=145.79.227.33:4871 , piece_id=7MAICINKRVUOR2EEAHBMTXRNGKHBXX3HO37THFRETEBX7FIYKSJQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 06:12:52.121923882 (590h43m51.993006917s) [noise=true , address=145.79.227.33:4871 , piece_id=7MAICINKRVUOR2EEAHBMTXRNGKHBXX3HO37THFRETEBX7FIYKSJQ, seg_pos={0 0}, upload=517860]: Long tail cancellation due to optimal threshold Dec 30 06:12:52.121942137 (590h43m51.993025172s) [noise=true , address=77.90.31.12:4589 , piece_id=T5PFZHHONH6ZU336I2HQZGD2E7OHR5RVH3MDLASSRJUYCLJTD4EQ, seg_pos={0 0}, upload=517860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 30 06:12:52.121949812 (590h43m51.993032857s) [noise=true , address=77.90.31.12:4589 , piece_id=T5PFZHHONH6ZU336I2HQZGD2E7OHR5RVH3MDLASSRJUYCLJTD4EQ, seg_pos={0 0}, upload=517860]: Long tail cancellation due to optimal threshold Dec 30 06:12:52.121967816 (590h43m51.993050851s) [seg_pos={0 0}, upload=517860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:12:52.122056395 (590h43m51.99313943s) [upload=517860]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:12:52.122064029 (590h43m51.993147064s) [upload=517860]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:12:52.412694056 (590h43m52.283777111s) [upload=517860]: Done uploading segments... Dec 30 06:12:52.412721609 (590h43m52.283804654s) [upload=517860]: Upload finished. err: <nil> Duration: 30.002883178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 40.463869982s trace: https://jaeger.dev.storj.io/trace/63ad9d48e40bce35 FAIL (2025-12-30 06:13:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143357498 (590h38m20.014440563s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143876246 (590h38m20.014959281s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144161779 (590h38m20.015244815s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144434238 (590h38m20.015517273s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.145849753 (590h38m20.016932908s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146180884 (590h38m20.017263919s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146593781 (590h38m20.017676826s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146881829 (590h38m20.017964864s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.147950344 (590h38m20.019033389s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.148036527 (590h38m20.019119582s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149117375 (590h38m20.020200431s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149235281 (590h38m20.020318316s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.150397153 (590h38m20.021480188s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151728988 (590h38m20.022812023s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151835581 (590h38m20.022918616s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:21.542337743 (590h38m21.413420788s) [upload=517783]: Next returned nil segment Dec 30 06:07:21.542350527 (590h38m21.413433572s) [upload=517783]: Waiting for error group managing segments... Dec 30 06:07:21.896542947 (590h38m21.767625982s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.011822447 (590h38m21.882905482s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.020838726 (590h38m21.891921761s) [noise=true , address=145.79.227.29:4966 , piece_id=XWMEKLXHRAPWDHAHOEL26J6KH7Y5QA5KGC3BIKBMQLPQXLPDOXNQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.021789667 (590h38m21.892872712s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.142912785 (590h38m22.01399583s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.167424118 (590h38m22.038507164s) [noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.178641458 (590h38m22.049724493s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.219274871 (590h38m22.090357906s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.222614600 (590h38m22.093697635s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.613549163 (590h38m22.484632208s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.893164776 (590h38m22.764247811s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967338144 (590h38m22.838421179s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967375515 (590h38m22.83845855s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972752413 (590h38m22.843835448s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972766070 (590h38m22.843849115s) [seg_pos={0 0}, upload=517783]: Segment reached optimal threshold of 14 pieces. Dec 30 06:07:22.972949789 (590h38m22.844032824s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ShGXckiJ3N5UWVmsMdhj98oFaRN7sg1KRucyUzqENW4Jp4kS): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.972956632 (590h38m22.844039667s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.972996318 (590h38m22.844079353s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973004804 (590h38m22.84408784s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973059238 (590h38m22.844142273s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973065109 (590h38m22.844148144s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973191119 (590h38m22.844274154s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection Dec 30 06:07:22.973197110 (590h38m22.844280145s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973306679 (590h38m22.844389714s) [seg_pos={0 0}, upload=517783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:07:22.973383645 (590h38m22.84446668s) [upload=517783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:07:22.973389166 (590h38m22.844472201s) [upload=517783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:07:23.274918237 (590h38m23.146001272s) [upload=517783]: Done uploading segments... Dec 30 06:07:23.274940459 (590h38m23.146023495s) [upload=517783]: Upload finished. err: <nil> Duration: 40.464007704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................X...........X] duration: 5.001559739s trace: https://jaeger.dev.storj.io/trace/19740e6f10941c1 FAIL (2025-12-30 06:13:00 UTC) orbiter warning: context deadline exceeded Duration: 5.001586811s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 34.038589412s trace: https://jaeger.dev.storj.io/trace/63ad9d48e40bce35 FAIL (2025-12-30 06:12:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143357498 (590h38m20.014440563s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143876246 (590h38m20.014959281s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144161779 (590h38m20.015244815s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144434238 (590h38m20.015517273s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.145849753 (590h38m20.016932908s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146180884 (590h38m20.017263919s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146593781 (590h38m20.017676826s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146881829 (590h38m20.017964864s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.147950344 (590h38m20.019033389s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.148036527 (590h38m20.019119582s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149117375 (590h38m20.020200431s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149235281 (590h38m20.020318316s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.150397153 (590h38m20.021480188s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151728988 (590h38m20.022812023s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151835581 (590h38m20.022918616s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:21.542337743 (590h38m21.413420788s) [upload=517783]: Next returned nil segment Dec 30 06:07:21.542350527 (590h38m21.413433572s) [upload=517783]: Waiting for error group managing segments... Dec 30 06:07:21.896542947 (590h38m21.767625982s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.011822447 (590h38m21.882905482s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.020838726 (590h38m21.891921761s) [noise=true , address=145.79.227.29:4966 , piece_id=XWMEKLXHRAPWDHAHOEL26J6KH7Y5QA5KGC3BIKBMQLPQXLPDOXNQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.021789667 (590h38m21.892872712s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.142912785 (590h38m22.01399583s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.167424118 (590h38m22.038507164s) [noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.178641458 (590h38m22.049724493s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.219274871 (590h38m22.090357906s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.222614600 (590h38m22.093697635s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.613549163 (590h38m22.484632208s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.893164776 (590h38m22.764247811s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967338144 (590h38m22.838421179s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967375515 (590h38m22.83845855s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972752413 (590h38m22.843835448s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972766070 (590h38m22.843849115s) [seg_pos={0 0}, upload=517783]: Segment reached optimal threshold of 14 pieces. Dec 30 06:07:22.972949789 (590h38m22.844032824s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ShGXckiJ3N5UWVmsMdhj98oFaRN7sg1KRucyUzqENW4Jp4kS): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.972956632 (590h38m22.844039667s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.972996318 (590h38m22.844079353s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973004804 (590h38m22.84408784s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973059238 (590h38m22.844142273s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973065109 (590h38m22.844148144s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973191119 (590h38m22.844274154s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection Dec 30 06:07:22.973197110 (590h38m22.844280145s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973306679 (590h38m22.844389714s) [seg_pos={0 0}, upload=517783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:07:22.973383645 (590h38m22.84446668s) [upload=517783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:07:22.973389166 (590h38m22.844472201s) [upload=517783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:07:23.274918237 (590h38m23.146001272s) [upload=517783]: Done uploading segments... Dec 30 06:07:23.274940459 (590h38m23.146023495s) [upload=517783]: Upload finished. err: <nil> Duration: 34.038637313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.X.......X..X..X....X.XX.X..X..X..X..XX.......X..X......] duration: 29.331653711s trace: https://jaeger.dev.storj.io/trace/2294dfd878d83329 FAIL (2025-12-30 06:12:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 06:06:24.040063500 (590h37m23.911146545s) [upload=517770]: Starting upload Dec 30 06:06:24.040132762 (590h37m23.911215807s) [upload=517770]: Done starting upload Dec 30 06:06:24.040263911 (590h37m23.911346946s) [upload=517770]: Uploading segments... Dec 30 06:06:24.040371576 (590h37m23.911454611s) [upload=517770]: Got next segment. Inline: false Dec 30 06:06:24.040426581 (590h37m23.911509616s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:06:24.040432091 (590h37m23.911515136s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:06:24.286449407 (590h37m24.157532453s) [seg_pos={0 0}, upload=517770]: Begin upload segment... Dec 30 06:06:24.286506156 (590h37m24.157589191s) [seg_pos={0 0}, upload=517770]: Done begin upload segment. Dec 30 06:06:24.286526915 (590h37m24.15760995s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.286548857 (590h37m24.157631902s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.287530015 (590h37m24.15861306s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288177458 (590h37m24.159260493s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288728628 (590h37m24.159811673s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.289619123 (590h37m24.160702158s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.297967280 (590h37m24.169050335s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.298857034 (590h37m24.169940079s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299465613 (590h37m24.170548648s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299909688 (590h37m24.170992734s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.300325350 (590h37m24.171408395s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.301916008 (590h37m24.172999043s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.303685588 (590h37m24.174768633s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.305653856 (590h37m24.176736891s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.307169682 (590h37m24.178252717s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.308970161 (590h37m24.180053196s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:25.994701949 (590h37m25.865784994s) [upload=517770]: Next returned nil segment Dec 30 06:06:25.994719962 (590h37m25.865803008s) [upload=517770]: Waiting for error group managing segments... Dec 30 06:06:26.281883259 (590h37m26.152966294s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.288719125 (590h37m26.15980216s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.303475884 (590h37m26.17455893s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.304426785 (590h37m26.17550983s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.307556575 (590h37m26.17863962s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.308508046 (590h37m26.179591091s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.322472406 (590h37m26.193555441s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.381641885 (590h37m26.25272493s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.387162747 (590h37m26.258245792s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.488505349 (590h37m26.359588384s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.589330827 (590h37m26.460413872s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.617391439 (590h37m26.488474484s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737865402 (590h37m26.608948447s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737890169 (590h37m26.608973204s) [seg_pos={0 0}, upload=517770]: Segment reached optimal threshold of 13 pieces. Dec 30 06:06:26.738110679 (590h37m26.609193714s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection Dec 30 06:06:26.738120557 (590h37m26.609203602s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738310900 (590h37m26.609393945s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738325928 (590h37m26.609408973s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738487045 (590h37m26.60957009s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738493698 (590h37m26.609576733s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738509808 (590h37m26.609592853s) [seg_pos={0 0}, upload=517770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:06:26.738592045 (590h37m26.60967508s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:06:26.738597195 (590h37m26.60968023s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:06:27.040735666 (590h37m26.911818721s) [upload=517770]: Done uploading segments... Dec 30 06:06:27.040759581 (590h37m26.911842626s) [upload=517770]: Upload finished. err: <nil> Duration: 29.331680452s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.606593969s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:12:04 UTC) orbiter warning: context deadline exceeded Duration: 3.606621691s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.467887729s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:11:36 UTC) orbiter warning: context deadline exceeded Duration: 3.467918388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X...X....] duration: 15.798830415s trace: https://jaeger.dev.storj.io/trace/63ad9d48e40bce35 FAIL (2025-12-30 06:11:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143357498 (590h38m20.014440563s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143876246 (590h38m20.014959281s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144161779 (590h38m20.015244815s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144434238 (590h38m20.015517273s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.145849753 (590h38m20.016932908s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146180884 (590h38m20.017263919s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146593781 (590h38m20.017676826s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146881829 (590h38m20.017964864s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.147950344 (590h38m20.019033389s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.148036527 (590h38m20.019119582s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149117375 (590h38m20.020200431s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149235281 (590h38m20.020318316s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.150397153 (590h38m20.021480188s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151728988 (590h38m20.022812023s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151835581 (590h38m20.022918616s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:21.542337743 (590h38m21.413420788s) [upload=517783]: Next returned nil segment Dec 30 06:07:21.542350527 (590h38m21.413433572s) [upload=517783]: Waiting for error group managing segments... Dec 30 06:07:21.896542947 (590h38m21.767625982s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.011822447 (590h38m21.882905482s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.020838726 (590h38m21.891921761s) [noise=true , address=145.79.227.29:4966 , piece_id=XWMEKLXHRAPWDHAHOEL26J6KH7Y5QA5KGC3BIKBMQLPQXLPDOXNQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.021789667 (590h38m21.892872712s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.142912785 (590h38m22.01399583s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.167424118 (590h38m22.038507164s) [noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.178641458 (590h38m22.049724493s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.219274871 (590h38m22.090357906s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.222614600 (590h38m22.093697635s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.613549163 (590h38m22.484632208s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.893164776 (590h38m22.764247811s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967338144 (590h38m22.838421179s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967375515 (590h38m22.83845855s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972752413 (590h38m22.843835448s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972766070 (590h38m22.843849115s) [seg_pos={0 0}, upload=517783]: Segment reached optimal threshold of 14 pieces. Dec 30 06:07:22.972949789 (590h38m22.844032824s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ShGXckiJ3N5UWVmsMdhj98oFaRN7sg1KRucyUzqENW4Jp4kS): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.972956632 (590h38m22.844039667s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.972996318 (590h38m22.844079353s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973004804 (590h38m22.84408784s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973059238 (590h38m22.844142273s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973065109 (590h38m22.844148144s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973191119 (590h38m22.844274154s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection Dec 30 06:07:22.973197110 (590h38m22.844280145s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973306679 (590h38m22.844389714s) [seg_pos={0 0}, upload=517783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:07:22.973383645 (590h38m22.84446668s) [upload=517783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:07:22.973389166 (590h38m22.844472201s) [upload=517783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:07:23.274918237 (590h38m23.146001272s) [upload=517783]: Done uploading segments... Dec 30 06:07:23.274940459 (590h38m23.146023495s) [upload=517783]: Upload finished. err: <nil> Duration: 15.798864359s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.44593676s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:11:18 UTC) orbiter warning: context deadline exceeded Duration: 3.445967769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.X.......X..X..X....X.XX.X..X..X..X..XX.......X..X.....] duration: 5.81595849s trace: https://jaeger.dev.storj.io/trace/2294dfd878d83329 FAIL (2025-12-30 06:10:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 06:06:24.040063500 (590h37m23.911146545s) [upload=517770]: Starting upload Dec 30 06:06:24.040132762 (590h37m23.911215807s) [upload=517770]: Done starting upload Dec 30 06:06:24.040263911 (590h37m23.911346946s) [upload=517770]: Uploading segments... Dec 30 06:06:24.040371576 (590h37m23.911454611s) [upload=517770]: Got next segment. Inline: false Dec 30 06:06:24.040426581 (590h37m23.911509616s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:06:24.040432091 (590h37m23.911515136s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:06:24.286449407 (590h37m24.157532453s) [seg_pos={0 0}, upload=517770]: Begin upload segment... Dec 30 06:06:24.286506156 (590h37m24.157589191s) [seg_pos={0 0}, upload=517770]: Done begin upload segment. Dec 30 06:06:24.286526915 (590h37m24.15760995s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.286548857 (590h37m24.157631902s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.287530015 (590h37m24.15861306s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288177458 (590h37m24.159260493s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288728628 (590h37m24.159811673s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.289619123 (590h37m24.160702158s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.297967280 (590h37m24.169050335s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.298857034 (590h37m24.169940079s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299465613 (590h37m24.170548648s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299909688 (590h37m24.170992734s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.300325350 (590h37m24.171408395s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.301916008 (590h37m24.172999043s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.303685588 (590h37m24.174768633s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.305653856 (590h37m24.176736891s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.307169682 (590h37m24.178252717s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.308970161 (590h37m24.180053196s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:25.994701949 (590h37m25.865784994s) [upload=517770]: Next returned nil segment Dec 30 06:06:25.994719962 (590h37m25.865803008s) [upload=517770]: Waiting for error group managing segments... Dec 30 06:06:26.281883259 (590h37m26.152966294s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.288719125 (590h37m26.15980216s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.303475884 (590h37m26.17455893s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.304426785 (590h37m26.17550983s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.307556575 (590h37m26.17863962s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.308508046 (590h37m26.179591091s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.322472406 (590h37m26.193555441s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.381641885 (590h37m26.25272493s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.387162747 (590h37m26.258245792s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.488505349 (590h37m26.359588384s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.589330827 (590h37m26.460413872s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.617391439 (590h37m26.488474484s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737865402 (590h37m26.608948447s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737890169 (590h37m26.608973204s) [seg_pos={0 0}, upload=517770]: Segment reached optimal threshold of 13 pieces. Dec 30 06:06:26.738110679 (590h37m26.609193714s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection Dec 30 06:06:26.738120557 (590h37m26.609203602s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738310900 (590h37m26.609393945s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738325928 (590h37m26.609408973s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738487045 (590h37m26.60957009s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738493698 (590h37m26.609576733s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738509808 (590h37m26.609592853s) [seg_pos={0 0}, upload=517770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:06:26.738592045 (590h37m26.60967508s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:06:26.738597195 (590h37m26.60968023s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:06:27.040735666 (590h37m26.911818721s) [upload=517770]: Done uploading segments... Dec 30 06:06:27.040759581 (590h37m26.911842626s) [upload=517770]: Upload finished. err: <nil> Duration: 5.815990761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 15.244206049s trace: https://jaeger.dev.storj.io/trace/63ad9d48e40bce35 FAIL (2025-12-30 06:10:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143357498 (590h38m20.014440563s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143876246 (590h38m20.014959281s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144161779 (590h38m20.015244815s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144434238 (590h38m20.015517273s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.145849753 (590h38m20.016932908s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146180884 (590h38m20.017263919s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146593781 (590h38m20.017676826s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146881829 (590h38m20.017964864s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.147950344 (590h38m20.019033389s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.148036527 (590h38m20.019119582s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149117375 (590h38m20.020200431s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149235281 (590h38m20.020318316s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.150397153 (590h38m20.021480188s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151728988 (590h38m20.022812023s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151835581 (590h38m20.022918616s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:21.542337743 (590h38m21.413420788s) [upload=517783]: Next returned nil segment Dec 30 06:07:21.542350527 (590h38m21.413433572s) [upload=517783]: Waiting for error group managing segments... Dec 30 06:07:21.896542947 (590h38m21.767625982s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.011822447 (590h38m21.882905482s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.020838726 (590h38m21.891921761s) [noise=true , address=145.79.227.29:4966 , piece_id=XWMEKLXHRAPWDHAHOEL26J6KH7Y5QA5KGC3BIKBMQLPQXLPDOXNQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.021789667 (590h38m21.892872712s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.142912785 (590h38m22.01399583s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.167424118 (590h38m22.038507164s) [noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.178641458 (590h38m22.049724493s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.219274871 (590h38m22.090357906s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.222614600 (590h38m22.093697635s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.613549163 (590h38m22.484632208s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.893164776 (590h38m22.764247811s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967338144 (590h38m22.838421179s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967375515 (590h38m22.83845855s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972752413 (590h38m22.843835448s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972766070 (590h38m22.843849115s) [seg_pos={0 0}, upload=517783]: Segment reached optimal threshold of 14 pieces. Dec 30 06:07:22.972949789 (590h38m22.844032824s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ShGXckiJ3N5UWVmsMdhj98oFaRN7sg1KRucyUzqENW4Jp4kS): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.972956632 (590h38m22.844039667s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.972996318 (590h38m22.844079353s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973004804 (590h38m22.84408784s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973059238 (590h38m22.844142273s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973065109 (590h38m22.844148144s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973191119 (590h38m22.844274154s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection Dec 30 06:07:22.973197110 (590h38m22.844280145s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973306679 (590h38m22.844389714s) [seg_pos={0 0}, upload=517783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:07:22.973383645 (590h38m22.84446668s) [upload=517783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:07:22.973389166 (590h38m22.844472201s) [upload=517783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:07:23.274918237 (590h38m23.146001272s) [upload=517783]: Done uploading segments... Dec 30 06:07:23.274940459 (590h38m23.146023495s) [upload=517783]: Upload finished. err: <nil> Duration: 15.244224323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.500055413s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:10:04 UTC) orbiter warning: context deadline exceeded Duration: 3.500079218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......X..X....] duration: 6.286688076s trace: https://jaeger.dev.storj.io/trace/2294dfd878d83329 FAIL (2025-12-30 06:10:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 06:06:24.040063500 (590h37m23.911146545s) [upload=517770]: Starting upload Dec 30 06:06:24.040132762 (590h37m23.911215807s) [upload=517770]: Done starting upload Dec 30 06:06:24.040263911 (590h37m23.911346946s) [upload=517770]: Uploading segments... Dec 30 06:06:24.040371576 (590h37m23.911454611s) [upload=517770]: Got next segment. Inline: false Dec 30 06:06:24.040426581 (590h37m23.911509616s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:06:24.040432091 (590h37m23.911515136s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:06:24.286449407 (590h37m24.157532453s) [seg_pos={0 0}, upload=517770]: Begin upload segment... Dec 30 06:06:24.286506156 (590h37m24.157589191s) [seg_pos={0 0}, upload=517770]: Done begin upload segment. Dec 30 06:06:24.286526915 (590h37m24.15760995s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.286548857 (590h37m24.157631902s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.287530015 (590h37m24.15861306s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288177458 (590h37m24.159260493s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288728628 (590h37m24.159811673s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.289619123 (590h37m24.160702158s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.297967280 (590h37m24.169050335s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.298857034 (590h37m24.169940079s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299465613 (590h37m24.170548648s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299909688 (590h37m24.170992734s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.300325350 (590h37m24.171408395s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.301916008 (590h37m24.172999043s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.303685588 (590h37m24.174768633s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.305653856 (590h37m24.176736891s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.307169682 (590h37m24.178252717s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.308970161 (590h37m24.180053196s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:25.994701949 (590h37m25.865784994s) [upload=517770]: Next returned nil segment Dec 30 06:06:25.994719962 (590h37m25.865803008s) [upload=517770]: Waiting for error group managing segments... Dec 30 06:06:26.281883259 (590h37m26.152966294s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.288719125 (590h37m26.15980216s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.303475884 (590h37m26.17455893s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.304426785 (590h37m26.17550983s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.307556575 (590h37m26.17863962s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.308508046 (590h37m26.179591091s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.322472406 (590h37m26.193555441s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.381641885 (590h37m26.25272493s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.387162747 (590h37m26.258245792s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.488505349 (590h37m26.359588384s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.589330827 (590h37m26.460413872s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.617391439 (590h37m26.488474484s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737865402 (590h37m26.608948447s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737890169 (590h37m26.608973204s) [seg_pos={0 0}, upload=517770]: Segment reached optimal threshold of 13 pieces. Dec 30 06:06:26.738110679 (590h37m26.609193714s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection Dec 30 06:06:26.738120557 (590h37m26.609203602s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738310900 (590h37m26.609393945s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738325928 (590h37m26.609408973s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738487045 (590h37m26.60957009s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738493698 (590h37m26.609576733s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738509808 (590h37m26.609592853s) [seg_pos={0 0}, upload=517770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:06:26.738592045 (590h37m26.60967508s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:06:26.738597195 (590h37m26.60968023s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:06:27.040735666 (590h37m26.911818721s) [upload=517770]: Done uploading segments... Dec 30 06:06:27.040759581 (590h37m26.911842626s) [upload=517770]: Upload finished. err: <nil> Duration: 6.28671681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 41.105474417s trace: https://jaeger.dev.storj.io/trace/63ad9d48e40bce35 FAIL (2025-12-30 06:10:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143357498 (590h38m20.014440563s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143876246 (590h38m20.014959281s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144161779 (590h38m20.015244815s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144434238 (590h38m20.015517273s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.145849753 (590h38m20.016932908s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146180884 (590h38m20.017263919s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146593781 (590h38m20.017676826s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146881829 (590h38m20.017964864s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.147950344 (590h38m20.019033389s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.148036527 (590h38m20.019119582s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149117375 (590h38m20.020200431s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149235281 (590h38m20.020318316s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.150397153 (590h38m20.021480188s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151728988 (590h38m20.022812023s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151835581 (590h38m20.022918616s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:21.542337743 (590h38m21.413420788s) [upload=517783]: Next returned nil segment Dec 30 06:07:21.542350527 (590h38m21.413433572s) [upload=517783]: Waiting for error group managing segments... Dec 30 06:07:21.896542947 (590h38m21.767625982s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.011822447 (590h38m21.882905482s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.020838726 (590h38m21.891921761s) [noise=true , address=145.79.227.29:4966 , piece_id=XWMEKLXHRAPWDHAHOEL26J6KH7Y5QA5KGC3BIKBMQLPQXLPDOXNQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.021789667 (590h38m21.892872712s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.142912785 (590h38m22.01399583s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.167424118 (590h38m22.038507164s) [noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.178641458 (590h38m22.049724493s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.219274871 (590h38m22.090357906s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.222614600 (590h38m22.093697635s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.613549163 (590h38m22.484632208s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.893164776 (590h38m22.764247811s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967338144 (590h38m22.838421179s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967375515 (590h38m22.83845855s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972752413 (590h38m22.843835448s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972766070 (590h38m22.843849115s) [seg_pos={0 0}, upload=517783]: Segment reached optimal threshold of 14 pieces. Dec 30 06:07:22.972949789 (590h38m22.844032824s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ShGXckiJ3N5UWVmsMdhj98oFaRN7sg1KRucyUzqENW4Jp4kS): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.972956632 (590h38m22.844039667s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.972996318 (590h38m22.844079353s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973004804 (590h38m22.84408784s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973059238 (590h38m22.844142273s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973065109 (590h38m22.844148144s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973191119 (590h38m22.844274154s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection Dec 30 06:07:22.973197110 (590h38m22.844280145s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973306679 (590h38m22.844389714s) [seg_pos={0 0}, upload=517783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:07:22.973383645 (590h38m22.84446668s) [upload=517783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:07:22.973389166 (590h38m22.844472201s) [upload=517783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:07:23.274918237 (590h38m23.146001272s) [upload=517783]: Done uploading segments... Dec 30 06:07:23.274940459 (590h38m23.146023495s) [upload=517783]: Upload finished. err: <nil> Duration: 41.105509555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......X..X...] duration: 5.891791288s trace: https://jaeger.dev.storj.io/trace/2294dfd878d83329 FAIL (2025-12-30 06:08:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 06:06:24.040063500 (590h37m23.911146545s) [upload=517770]: Starting upload Dec 30 06:06:24.040132762 (590h37m23.911215807s) [upload=517770]: Done starting upload Dec 30 06:06:24.040263911 (590h37m23.911346946s) [upload=517770]: Uploading segments... Dec 30 06:06:24.040371576 (590h37m23.911454611s) [upload=517770]: Got next segment. Inline: false Dec 30 06:06:24.040426581 (590h37m23.911509616s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:06:24.040432091 (590h37m23.911515136s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:06:24.286449407 (590h37m24.157532453s) [seg_pos={0 0}, upload=517770]: Begin upload segment... Dec 30 06:06:24.286506156 (590h37m24.157589191s) [seg_pos={0 0}, upload=517770]: Done begin upload segment. Dec 30 06:06:24.286526915 (590h37m24.15760995s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.286548857 (590h37m24.157631902s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.287530015 (590h37m24.15861306s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288177458 (590h37m24.159260493s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288728628 (590h37m24.159811673s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.289619123 (590h37m24.160702158s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.297967280 (590h37m24.169050335s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.298857034 (590h37m24.169940079s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299465613 (590h37m24.170548648s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299909688 (590h37m24.170992734s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.300325350 (590h37m24.171408395s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.301916008 (590h37m24.172999043s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.303685588 (590h37m24.174768633s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.305653856 (590h37m24.176736891s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.307169682 (590h37m24.178252717s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.308970161 (590h37m24.180053196s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:25.994701949 (590h37m25.865784994s) [upload=517770]: Next returned nil segment Dec 30 06:06:25.994719962 (590h37m25.865803008s) [upload=517770]: Waiting for error group managing segments... Dec 30 06:06:26.281883259 (590h37m26.152966294s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.288719125 (590h37m26.15980216s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.303475884 (590h37m26.17455893s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.304426785 (590h37m26.17550983s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.307556575 (590h37m26.17863962s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.308508046 (590h37m26.179591091s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.322472406 (590h37m26.193555441s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.381641885 (590h37m26.25272493s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.387162747 (590h37m26.258245792s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.488505349 (590h37m26.359588384s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.589330827 (590h37m26.460413872s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.617391439 (590h37m26.488474484s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737865402 (590h37m26.608948447s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737890169 (590h37m26.608973204s) [seg_pos={0 0}, upload=517770]: Segment reached optimal threshold of 13 pieces. Dec 30 06:06:26.738110679 (590h37m26.609193714s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection Dec 30 06:06:26.738120557 (590h37m26.609203602s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738310900 (590h37m26.609393945s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738325928 (590h37m26.609408973s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738487045 (590h37m26.60957009s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738493698 (590h37m26.609576733s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738509808 (590h37m26.609592853s) [seg_pos={0 0}, upload=517770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:06:26.738592045 (590h37m26.60967508s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:06:26.738597195 (590h37m26.60968023s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:06:27.040735666 (590h37m26.911818721s) [upload=517770]: Done uploading segments... Dec 30 06:06:27.040759581 (590h37m26.911842626s) [upload=517770]: Upload finished. err: <nil> Duration: 5.891822877s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 4.946108822s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:08:54 UTC) orbiter warning: context deadline exceeded Duration: 4.946141194s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 26.412357653s trace: https://jaeger.dev.storj.io/trace/63ad9d48e40bce35 FAIL (2025-12-30 06:08:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143357498 (590h38m20.014440563s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143876246 (590h38m20.014959281s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144161779 (590h38m20.015244815s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144434238 (590h38m20.015517273s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.145849753 (590h38m20.016932908s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146180884 (590h38m20.017263919s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146593781 (590h38m20.017676826s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146881829 (590h38m20.017964864s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.147950344 (590h38m20.019033389s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.148036527 (590h38m20.019119582s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149117375 (590h38m20.020200431s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149235281 (590h38m20.020318316s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.150397153 (590h38m20.021480188s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151728988 (590h38m20.022812023s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151835581 (590h38m20.022918616s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:21.542337743 (590h38m21.413420788s) [upload=517783]: Next returned nil segment Dec 30 06:07:21.542350527 (590h38m21.413433572s) [upload=517783]: Waiting for error group managing segments... Dec 30 06:07:21.896542947 (590h38m21.767625982s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.011822447 (590h38m21.882905482s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.020838726 (590h38m21.891921761s) [noise=true , address=145.79.227.29:4966 , piece_id=XWMEKLXHRAPWDHAHOEL26J6KH7Y5QA5KGC3BIKBMQLPQXLPDOXNQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.021789667 (590h38m21.892872712s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.142912785 (590h38m22.01399583s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.167424118 (590h38m22.038507164s) [noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.178641458 (590h38m22.049724493s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.219274871 (590h38m22.090357906s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.222614600 (590h38m22.093697635s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.613549163 (590h38m22.484632208s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.893164776 (590h38m22.764247811s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967338144 (590h38m22.838421179s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967375515 (590h38m22.83845855s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972752413 (590h38m22.843835448s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972766070 (590h38m22.843849115s) [seg_pos={0 0}, upload=517783]: Segment reached optimal threshold of 14 pieces. Dec 30 06:07:22.972949789 (590h38m22.844032824s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ShGXckiJ3N5UWVmsMdhj98oFaRN7sg1KRucyUzqENW4Jp4kS): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.972956632 (590h38m22.844039667s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.972996318 (590h38m22.844079353s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973004804 (590h38m22.84408784s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973059238 (590h38m22.844142273s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973065109 (590h38m22.844148144s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973191119 (590h38m22.844274154s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection Dec 30 06:07:22.973197110 (590h38m22.844280145s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973306679 (590h38m22.844389714s) [seg_pos={0 0}, upload=517783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:07:22.973383645 (590h38m22.84446668s) [upload=517783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:07:22.973389166 (590h38m22.844472201s) [upload=517783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:07:23.274918237 (590h38m23.146001272s) [upload=517783]: Done uploading segments... Dec 30 06:07:23.274940459 (590h38m23.146023495s) [upload=517783]: Upload finished. err: <nil> Duration: 26.412371638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 59.362316198s trace: https://jaeger.dev.storj.io/trace/63ad9d48e40bce35 FAIL (2025-12-30 06:08:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143357498 (590h38m20.014440563s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.143876246 (590h38m20.014959281s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144161779 (590h38m20.015244815s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.144434238 (590h38m20.015517273s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.145849753 (590h38m20.016932908s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146180884 (590h38m20.017263919s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146593781 (590h38m20.017676826s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.146881829 (590h38m20.017964864s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.147950344 (590h38m20.019033389s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.148036527 (590h38m20.019119582s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149117375 (590h38m20.020200431s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.149235281 (590h38m20.020318316s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.150397153 (590h38m20.021480188s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151728988 (590h38m20.022812023s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:20.151835581 (590h38m20.022918616s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Uploading piece... Dec 30 06:07:21.542337743 (590h38m21.413420788s) [upload=517783]: Next returned nil segment Dec 30 06:07:21.542350527 (590h38m21.413433572s) [upload=517783]: Waiting for error group managing segments... Dec 30 06:07:21.896542947 (590h38m21.767625982s) [noise=true , address=64.49.13.154:4932 , piece_id=3P5RMVY7L7YSQBEUTCKHNTDVTK2XGXHD4J42FYEPZ55NFKU7BOWQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.011822447 (590h38m21.882905482s) [noise=true , address=145.79.227.35:4504 , piece_id=7VW6N7C7F37WSZYH7JSYRGLQOBQCOOF2VYR6YCC5XIPPNNZIP6OA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.020838726 (590h38m21.891921761s) [noise=true , address=145.79.227.29:4966 , piece_id=XWMEKLXHRAPWDHAHOEL26J6KH7Y5QA5KGC3BIKBMQLPQXLPDOXNQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.021789667 (590h38m21.892872712s) [noise=true , address=145.79.227.13:4815 , piece_id=5LQW2S47TTICFP5UCW3WMPUCQRGVINNON2MCCEIFIRS7X444EZ3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.142912785 (590h38m22.01399583s) [noise=true , address=145.79.227.34:4927 , piece_id=IM4YV3PZZ6P5JNMKEKHJ43HIEBU3FONPSVNYPE4FKD5BNUV35EXA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.167424118 (590h38m22.038507164s) [noise=true , address=77.90.31.17:4800 , piece_id=X26B4347XTQSITS3V3CTT5KZTMPSJ4IOC6E3RSF2OQC4E36UYRAA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.178641458 (590h38m22.049724493s) [noise=true , address=77.90.31.15:4706 , piece_id=Z2OCC3UMT5DEZAQJVBJ24EK4UD5ID7LI2YIMTIQYT3IYPOK6H3EA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.219274871 (590h38m22.090357906s) [noise=true , address=77.90.31.13:4656 , piece_id=LPIJVZSJWIR265N7RMEU2CCH3HLT2Z3KB5G7DDNJ4C4FZW4ZURFA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.222614600 (590h38m22.093697635s) [noise=true , address=77.90.31.11:4572 , piece_id=DFIWZ7XQTJUPQRGBQPPSYSFNVHRJKSHMAFLDBICJBL7QSWY7ZJLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.613549163 (590h38m22.484632208s) [noise=true , address=145.79.227.6:4263 , piece_id=AGT3H5LX5ZTKXSEFTVXEJHD3KQICZDEKIWDIU5JSCMOKFFVVOVKQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.893164776 (590h38m22.764247811s) [noise=true , address=145.79.227.44:4220 , piece_id=LFTKVP2GHZWEUK62MREY3BEUOO5OCRPQBKWQIM3Y6EDZXNQ6KWLA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967338144 (590h38m22.838421179s) [noise=true , address=145.79.227.43:4184 , piece_id=CKVEN67R6LZSX4ZFCB65J6YXMGXKDUMLREYH2LSN5LIHZPRR5JTQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.967375515 (590h38m22.83845855s) [noise=true , address=145.79.227.33:4900 , piece_id=F4SDPLNSGWO7WRP4TMUBZKBJRS7JXVUOW4SBDFCRFWDHIE2BWPYQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972752413 (590h38m22.843835448s) [noise=true , address=145.79.227.40:4068 , piece_id=4NKN4EVAQHDXTKZHYYTJFOMIUFH2FQE6BJA7JYXOH3VEZ43EKOIQ, seg_pos={0 0}, upload=517783]: Done uploading piece. err: <nil> Dec 30 06:07:22.972766070 (590h38m22.843849115s) [seg_pos={0 0}, upload=517783]: Segment reached optimal threshold of 14 pieces. Dec 30 06:07:22.972949789 (590h38m22.844032824s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ShGXckiJ3N5UWVmsMdhj98oFaRN7sg1KRucyUzqENW4Jp4kS): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.972956632 (590h38m22.844039667s) [noise=true , address=145.79.227.11:4735 , piece_id=E2GHILKDUFTJXLJT5BJ3XGZFT3G2WKPDVXIQBY7O2R2PG6O2XN3Q, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.972996318 (590h38m22.844079353s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973004804 (590h38m22.84408784s) [noise=true , address=145.79.227.30:4765 , piece_id=R5F5XRBQ54JIZC5LOYW7ZWQSQVEOAV6QFFH5BVLWCUMCXCTEOORA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973059238 (590h38m22.844142273s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YZmFUbHJ3DfhJ1fCqMcy2tdDvmdPogiNAiFwduvcDgsw1QAZQ): protocol: expected piece hash; context canceled; EOF Dec 30 06:07:22.973065109 (590h38m22.844148144s) [noise=true , address=45.91.116.10:4106 , piece_id=76PJA7PLOD6WMSBZA5XR6R7BOXDI5SUGVS5YEDTBGDTPMRMA6BRA, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973191119 (590h38m22.844274154s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36658->64.49.13.169:4707: use of closed network connection Dec 30 06:07:22.973197110 (590h38m22.844280145s) [noise=true , address=64.49.13.169:4707 , piece_id=OWFLXFTFQIY6UNQ5T25SUILUJEPR743DTV4PT6PS4MMEEBZFOA6A, seg_pos={0 0}, upload=517783]: Long tail cancellation due to optimal threshold Dec 30 06:07:22.973306679 (590h38m22.844389714s) [seg_pos={0 0}, upload=517783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:07:22.973383645 (590h38m22.84446668s) [upload=517783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:07:22.973389166 (590h38m22.844472201s) [upload=517783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:07:23.274918237 (590h38m23.146001272s) [upload=517783]: Done uploading segments... Dec 30 06:07:23.274940459 (590h38m23.146023495s) [upload=517783]: Upload finished. err: <nil> Duration: 59.362341986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......X..X..] duration: 6.09260745s trace: https://jaeger.dev.storj.io/trace/2294dfd878d83329 FAIL (2025-12-30 06:08:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 06:06:24.040063500 (590h37m23.911146545s) [upload=517770]: Starting upload Dec 30 06:06:24.040132762 (590h37m23.911215807s) [upload=517770]: Done starting upload Dec 30 06:06:24.040263911 (590h37m23.911346946s) [upload=517770]: Uploading segments... Dec 30 06:06:24.040371576 (590h37m23.911454611s) [upload=517770]: Got next segment. Inline: false Dec 30 06:06:24.040426581 (590h37m23.911509616s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:06:24.040432091 (590h37m23.911515136s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:06:24.286449407 (590h37m24.157532453s) [seg_pos={0 0}, upload=517770]: Begin upload segment... Dec 30 06:06:24.286506156 (590h37m24.157589191s) [seg_pos={0 0}, upload=517770]: Done begin upload segment. Dec 30 06:06:24.286526915 (590h37m24.15760995s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.286548857 (590h37m24.157631902s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.287530015 (590h37m24.15861306s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288177458 (590h37m24.159260493s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288728628 (590h37m24.159811673s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.289619123 (590h37m24.160702158s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.297967280 (590h37m24.169050335s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.298857034 (590h37m24.169940079s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299465613 (590h37m24.170548648s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299909688 (590h37m24.170992734s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.300325350 (590h37m24.171408395s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.301916008 (590h37m24.172999043s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.303685588 (590h37m24.174768633s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.305653856 (590h37m24.176736891s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.307169682 (590h37m24.178252717s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.308970161 (590h37m24.180053196s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:25.994701949 (590h37m25.865784994s) [upload=517770]: Next returned nil segment Dec 30 06:06:25.994719962 (590h37m25.865803008s) [upload=517770]: Waiting for error group managing segments... Dec 30 06:06:26.281883259 (590h37m26.152966294s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.288719125 (590h37m26.15980216s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.303475884 (590h37m26.17455893s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.304426785 (590h37m26.17550983s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.307556575 (590h37m26.17863962s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.308508046 (590h37m26.179591091s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.322472406 (590h37m26.193555441s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.381641885 (590h37m26.25272493s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.387162747 (590h37m26.258245792s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.488505349 (590h37m26.359588384s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.589330827 (590h37m26.460413872s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.617391439 (590h37m26.488474484s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737865402 (590h37m26.608948447s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737890169 (590h37m26.608973204s) [seg_pos={0 0}, upload=517770]: Segment reached optimal threshold of 13 pieces. Dec 30 06:06:26.738110679 (590h37m26.609193714s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection Dec 30 06:06:26.738120557 (590h37m26.609203602s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738310900 (590h37m26.609393945s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738325928 (590h37m26.609408973s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738487045 (590h37m26.60957009s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738493698 (590h37m26.609576733s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738509808 (590h37m26.609592853s) [seg_pos={0 0}, upload=517770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:06:26.738592045 (590h37m26.60967508s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:06:26.738597195 (590h37m26.60968023s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:06:27.040735666 (590h37m26.911818721s) [upload=517770]: Done uploading segments... Dec 30 06:06:27.040759581 (590h37m26.911842626s) [upload=517770]: Upload finished. err: <nil> Duration: 6.09263809s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.366280453s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:07:45 UTC) orbiter warning: context deadline exceeded Duration: 3.366304951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......X..X.] duration: 6.21276506s trace: https://jaeger.dev.storj.io/trace/2294dfd878d83329 FAIL (2025-12-30 06:06:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 06:06:24.040063500 (590h37m23.911146545s) [upload=517770]: Starting upload Dec 30 06:06:24.040132762 (590h37m23.911215807s) [upload=517770]: Done starting upload Dec 30 06:06:24.040263911 (590h37m23.911346946s) [upload=517770]: Uploading segments... Dec 30 06:06:24.040371576 (590h37m23.911454611s) [upload=517770]: Got next segment. Inline: false Dec 30 06:06:24.040426581 (590h37m23.911509616s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:06:24.040432091 (590h37m23.911515136s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:06:24.286449407 (590h37m24.157532453s) [seg_pos={0 0}, upload=517770]: Begin upload segment... Dec 30 06:06:24.286506156 (590h37m24.157589191s) [seg_pos={0 0}, upload=517770]: Done begin upload segment. Dec 30 06:06:24.286526915 (590h37m24.15760995s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.286548857 (590h37m24.157631902s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.287530015 (590h37m24.15861306s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288177458 (590h37m24.159260493s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288728628 (590h37m24.159811673s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.289619123 (590h37m24.160702158s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.297967280 (590h37m24.169050335s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.298857034 (590h37m24.169940079s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299465613 (590h37m24.170548648s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299909688 (590h37m24.170992734s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.300325350 (590h37m24.171408395s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.301916008 (590h37m24.172999043s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.303685588 (590h37m24.174768633s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.305653856 (590h37m24.176736891s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.307169682 (590h37m24.178252717s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.308970161 (590h37m24.180053196s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:25.994701949 (590h37m25.865784994s) [upload=517770]: Next returned nil segment Dec 30 06:06:25.994719962 (590h37m25.865803008s) [upload=517770]: Waiting for error group managing segments... Dec 30 06:06:26.281883259 (590h37m26.152966294s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.288719125 (590h37m26.15980216s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.303475884 (590h37m26.17455893s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.304426785 (590h37m26.17550983s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.307556575 (590h37m26.17863962s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.308508046 (590h37m26.179591091s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.322472406 (590h37m26.193555441s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.381641885 (590h37m26.25272493s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.387162747 (590h37m26.258245792s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.488505349 (590h37m26.359588384s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.589330827 (590h37m26.460413872s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.617391439 (590h37m26.488474484s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737865402 (590h37m26.608948447s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737890169 (590h37m26.608973204s) [seg_pos={0 0}, upload=517770]: Segment reached optimal threshold of 13 pieces. Dec 30 06:06:26.738110679 (590h37m26.609193714s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection Dec 30 06:06:26.738120557 (590h37m26.609203602s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738310900 (590h37m26.609393945s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738325928 (590h37m26.609408973s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738487045 (590h37m26.60957009s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738493698 (590h37m26.609576733s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738509808 (590h37m26.609592853s) [seg_pos={0 0}, upload=517770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:06:26.738592045 (590h37m26.60967508s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:06:26.738597195 (590h37m26.60968023s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:06:27.040735666 (590h37m26.911818721s) [upload=517770]: Done uploading segments... Dec 30 06:06:27.040759581 (590h37m26.911842626s) [upload=517770]: Upload finished. err: <nil> Duration: 6.21280172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......X..X] duration: 30.002234056s trace: https://jaeger.dev.storj.io/trace/2294dfd878d83329 FAIL (2025-12-30 06:06:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 06:06:24.040063500 (590h37m23.911146545s) [upload=517770]: Starting upload Dec 30 06:06:24.040132762 (590h37m23.911215807s) [upload=517770]: Done starting upload Dec 30 06:06:24.040263911 (590h37m23.911346946s) [upload=517770]: Uploading segments... Dec 30 06:06:24.040371576 (590h37m23.911454611s) [upload=517770]: Got next segment. Inline: false Dec 30 06:06:24.040426581 (590h37m23.911509616s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:06:24.040432091 (590h37m23.911515136s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:06:24.286449407 (590h37m24.157532453s) [seg_pos={0 0}, upload=517770]: Begin upload segment... Dec 30 06:06:24.286506156 (590h37m24.157589191s) [seg_pos={0 0}, upload=517770]: Done begin upload segment. Dec 30 06:06:24.286526915 (590h37m24.15760995s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.286548857 (590h37m24.157631902s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.287530015 (590h37m24.15861306s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288177458 (590h37m24.159260493s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.288728628 (590h37m24.159811673s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.289619123 (590h37m24.160702158s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.297967280 (590h37m24.169050335s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.298857034 (590h37m24.169940079s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299465613 (590h37m24.170548648s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.299909688 (590h37m24.170992734s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.300325350 (590h37m24.171408395s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.301916008 (590h37m24.172999043s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.303685588 (590h37m24.174768633s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.305653856 (590h37m24.176736891s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.307169682 (590h37m24.178252717s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:24.308970161 (590h37m24.180053196s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Uploading piece... Dec 30 06:06:25.994701949 (590h37m25.865784994s) [upload=517770]: Next returned nil segment Dec 30 06:06:25.994719962 (590h37m25.865803008s) [upload=517770]: Waiting for error group managing segments... Dec 30 06:06:26.281883259 (590h37m26.152966294s) [noise=true , address=145.79.227.38:4977 , piece_id=YKSIUJ6FMSZZ2LNZNZFSTJVLNUB54WJRUFIM6PNC2WDW6VLZ3ASQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.288719125 (590h37m26.15980216s) [noise=true , address=145.79.227.9:4367 , piece_id=U5R6AV6QLCK6EFQXLMRSPYLVQNBYFM36BPF77HKIWU7DARDYV7ZA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.303475884 (590h37m26.17455893s) [noise=true , address=145.79.227.48:4302 , piece_id=6EWH3D4YU23EJZAWUMYMAZ5CSLBT2CCYLUNG5FTH7426RXR3IBKQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.304426785 (590h37m26.17550983s) [noise=true , address=145.79.227.25:4619 , piece_id=S6EQ3C5RLZ45XK52T2TJVC5P47XWG5MU44WVM46J4ZFUEEWXGEAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.307556575 (590h37m26.17863962s) [noise=true , address=145.79.227.7:4293 , piece_id=LN7MIDHNZ7BZDW3ERJNCWZJT7BGGHC2C4VAFPHY4ZYCB2YQVPOAQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.308508046 (590h37m26.179591091s) [noise=true , address=145.79.227.5:4221 , piece_id=T5S7J3ECSIFWKOLLT2DZNVO6CXADL4DVDV4A6YFAVYTLUHXMPYJA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.322472406 (590h37m26.193555441s) [noise=true , address=145.79.227.31:4798 , piece_id=HG25MAOCWHELRR4SGBELXI6PDWACUJYVXZFUKDKXYMJJHOHXYPYQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.381641885 (590h37m26.25272493s) [noise=true , address=145.79.227.19:4077 , piece_id=U5OLEKVAB4PZSBYDUMIRXYFRZAYTCLWITQ55K5ZKPCS4XPKRRG6Q, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.387162747 (590h37m26.258245792s) [noise=true , address=145.79.227.41:4087 , piece_id=NXSSZJBQKX347Y777RGXNM7C6CXKMPDMVVZLVDWUW2PUK4KJVYIA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.488505349 (590h37m26.359588384s) [noise=true , address=77.90.31.17:4773 , piece_id=TK2LB6YDYOY2NYZBWMRUUF236SB3MZQJUGL23TV6USIXKKVEBIHQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.589330827 (590h37m26.460413872s) [noise=true , address=77.90.31.14:4663 , piece_id=J7VLEDYSYI2IR4XF5VFYA2HV45GAESCFRBXMBQTXJMTSZUDEW6DQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.617391439 (590h37m26.488474484s) [noise=true , address=45.91.116.4:4937 , piece_id=Z4NRKX4EHBT4IS2TLXPWR4YWEB6NE365X5VOGCZPBZWJIFGBHIMA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737865402 (590h37m26.608948447s) [noise=true , address=45.91.116.12:4228 , piece_id=AEKYZRDHMCL5VJGF4SA3EGNXWP2XE3JLUOKBWXWZHLIUEW7BRHEQ, seg_pos={0 0}, upload=517770]: Done uploading piece. err: <nil> Dec 30 06:06:26.737890169 (590h37m26.608973204s) [seg_pos={0 0}, upload=517770]: Segment reached optimal threshold of 13 pieces. Dec 30 06:06:26.738110679 (590h37m26.609193714s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46048->77.90.31.11:4556: use of closed network connection Dec 30 06:06:26.738120557 (590h37m26.609203602s) [noise=true , address=77.90.31.11:4556 , piece_id=UP62EBKBEOA4654ZT7LKL7CO6FMVN3DXLLC6POMIUKV6PT6MTOXA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738310900 (590h37m26.609393945s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738325928 (590h37m26.609408973s) [noise=true , address=45.91.116.7:4009 , piece_id=A5HYGLQX4YUMC36KYGNJSLDSXBM6RBS335DHLAUT2NWIZ4HYOTKA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738487045 (590h37m26.60957009s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 06:06:26.738493698 (590h37m26.609576733s) [noise=true , address=45.91.116.9:4262 , piece_id=ORF4ZG4BW4AKF46RL3M5M3UDEI6AZPMF7UXM57JDRCQGBRR3R7YA, seg_pos={0 0}, upload=517770]: Long tail cancellation due to optimal threshold Dec 30 06:06:26.738509808 (590h37m26.609592853s) [seg_pos={0 0}, upload=517770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:06:26.738592045 (590h37m26.60967508s) [upload=517770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:06:26.738597195 (590h37m26.60968023s) [upload=517770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:06:27.040735666 (590h37m26.911818721s) [upload=517770]: Done uploading segments... Dec 30 06:06:27.040759581 (590h37m26.911842626s) [upload=517770]: Upload finished. err: <nil> Duration: 30.002254635s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.600590557s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:06:48 UTC) orbiter warning: context deadline exceeded Duration: 3.600614081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 15.355791267s trace: https://jaeger.dev.storj.io/trace/29aa707bf8bd3de6 FAIL (2025-12-30 06:06:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.517379430 (590h34m25.388462475s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.519724064 (590h34m25.39080711s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.522068399 (590h34m25.393151434s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.523362533 (590h34m25.394445578s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.524404988 (590h34m25.395488023s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.525938748 (590h34m25.397021793s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.526464530 (590h34m25.397547575s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.529259672 (590h34m25.400342717s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.531769822 (590h34m25.402852867s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.532925572 (590h34m25.404008607s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.534415720 (590h34m25.405498765s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537220020 (590h34m25.408303065s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537490615 (590h34m25.40857365s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.539400482 (590h34m25.410483517s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.541128883 (590h34m25.412211918s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:26.991260789 (590h34m26.862343834s) [upload=517728]: Next returned nil segment Dec 30 06:03:26.991297679 (590h34m26.862380734s) [upload=517728]: Waiting for error group managing segments... Dec 30 06:03:27.434491232 (590h34m27.305574277s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.473885785 (590h34m27.34496882s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.481026933 (590h34m27.352109968s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.537133078 (590h34m27.408216113s) [noise=true , address=145.79.227.43:4174 , piece_id=PYWCVSCRU45AHKUIGBP6XXOQSR7CS5GIU2W3CV2C5VIAVJAFMYAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.610086884 (590h34m27.481169919s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682091743 (590h34m27.553174788s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682634627 (590h34m27.553717672s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.712364247 (590h34m27.583447292s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.000168182 (590h34m27.871251227s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.289616990 (590h34m28.160700025s) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.306681834 (590h34m28.177764879s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.312426353 (590h34m28.183509388s) [noise=true , address=145.79.227.30:4779 , piece_id=KBETI2JQ3NDRDFTGFA5FW7S5IYL2TFIZ3NXTKQZX7JZQ5RTMX2LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.349481942 (590h34m28.220564987s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380570801 (590h34m28.251653836s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380591970 (590h34m28.251675016s) [seg_pos={0 0}, upload=517728]: Segment reached optimal threshold of 14 pieces. Dec 30 06:03:28.380822891 (590h34m28.251905926s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380830565 (590h34m28.2519136s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380925726 (590h34m28.252008771s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection Dec 30 06:03:28.380942197 (590h34m28.252025242s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380958769 (590h34m28.252041814s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380967896 (590h34m28.252050942s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381091813 (590h34m28.252174858s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.381098525 (590h34m28.252181561s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381144553 (590h34m28.252227588s) [seg_pos={0 0}, upload=517728]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:03:28.381225898 (590h34m28.252308933s) [upload=517728]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:03:28.381231308 (590h34m28.252314344s) [upload=517728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:03:28.673388913 (590h34m28.544471958s) [upload=517728]: Done uploading segments... Dec 30 06:03:28.673412569 (590h34m28.544495604s) [upload=517728]: Upload finished. err: <nil> Duration: 15.35581941s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.7664604s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:05:48 UTC) orbiter warning: context deadline exceeded Duration: 3.766486991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 27.614387679s trace: https://jaeger.dev.storj.io/trace/29aa707bf8bd3de6 FAIL (2025-12-30 06:05:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.517379430 (590h34m25.388462475s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.519724064 (590h34m25.39080711s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.522068399 (590h34m25.393151434s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.523362533 (590h34m25.394445578s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.524404988 (590h34m25.395488023s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.525938748 (590h34m25.397021793s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.526464530 (590h34m25.397547575s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.529259672 (590h34m25.400342717s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.531769822 (590h34m25.402852867s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.532925572 (590h34m25.404008607s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.534415720 (590h34m25.405498765s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537220020 (590h34m25.408303065s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537490615 (590h34m25.40857365s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.539400482 (590h34m25.410483517s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.541128883 (590h34m25.412211918s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:26.991260789 (590h34m26.862343834s) [upload=517728]: Next returned nil segment Dec 30 06:03:26.991297679 (590h34m26.862380734s) [upload=517728]: Waiting for error group managing segments... Dec 30 06:03:27.434491232 (590h34m27.305574277s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.473885785 (590h34m27.34496882s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.481026933 (590h34m27.352109968s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.537133078 (590h34m27.408216113s) [noise=true , address=145.79.227.43:4174 , piece_id=PYWCVSCRU45AHKUIGBP6XXOQSR7CS5GIU2W3CV2C5VIAVJAFMYAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.610086884 (590h34m27.481169919s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682091743 (590h34m27.553174788s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682634627 (590h34m27.553717672s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.712364247 (590h34m27.583447292s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.000168182 (590h34m27.871251227s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.289616990 (590h34m28.160700025s) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.306681834 (590h34m28.177764879s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.312426353 (590h34m28.183509388s) [noise=true , address=145.79.227.30:4779 , piece_id=KBETI2JQ3NDRDFTGFA5FW7S5IYL2TFIZ3NXTKQZX7JZQ5RTMX2LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.349481942 (590h34m28.220564987s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380570801 (590h34m28.251653836s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380591970 (590h34m28.251675016s) [seg_pos={0 0}, upload=517728]: Segment reached optimal threshold of 14 pieces. Dec 30 06:03:28.380822891 (590h34m28.251905926s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380830565 (590h34m28.2519136s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380925726 (590h34m28.252008771s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection Dec 30 06:03:28.380942197 (590h34m28.252025242s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380958769 (590h34m28.252041814s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380967896 (590h34m28.252050942s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381091813 (590h34m28.252174858s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.381098525 (590h34m28.252181561s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381144553 (590h34m28.252227588s) [seg_pos={0 0}, upload=517728]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:03:28.381225898 (590h34m28.252308933s) [upload=517728]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:03:28.381231308 (590h34m28.252314344s) [upload=517728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:03:28.673388913 (590h34m28.544471958s) [upload=517728]: Done uploading segments... Dec 30 06:03:28.673412569 (590h34m28.544495604s) [upload=517728]: Upload finished. err: <nil> Duration: 27.614413828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......X..] duration: 19.829983601s trace: https://jaeger.dev.storj.io/trace/1237689ed7bb48cb FAIL (2025-12-30 06:05:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA) context deadline exceeded ======LOGS====== Dec 30 06:03:43.669744944 (590h34m43.540827989s) [upload=517732]: Starting upload Dec 30 06:03:43.669856857 (590h34m43.540939902s) [upload=517732]: Done starting upload Dec 30 06:03:43.670047460 (590h34m43.541130515s) [upload=517732]: Uploading segments... Dec 30 06:03:43.670118325 (590h34m43.54120138s) [upload=517732]: Got next segment. Inline: false Dec 30 06:03:43.670156988 (590h34m43.541240063s) [upload=517732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:03:43.670163641 (590h34m43.541246686s) [upload=517732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:03:43.917142178 (590h34m43.788225213s) [seg_pos={0 0}, upload=517732]: Begin upload segment... Dec 30 06:03:43.917189789 (590h34m43.788272834s) [seg_pos={0 0}, upload=517732]: Done begin upload segment. Dec 30 06:03:43.917215047 (590h34m43.788298082s) [noise=true , address=145.79.227.14:4837 , piece_id=2RR2YOJE4Z2HPO662QZMAH4BWUJIVP26T3LUHM4N6Y3CQEPWDB7Q, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.918021964 (590h34m43.789105019s) [noise=true , address=145.79.227.15:4873 , piece_id=G2V6NLM4Q2JGZOB7CIX224JYFQUXPCQXR6YR3YZ7HU7QNP7LPQRA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919021606 (590h34m43.790104651s) [noise=true , address=77.90.31.14:4664 , piece_id=V6BZMBCU5Z6ZKUXNZ7E7AGRTKUNTMDEA3532BVP5K2DJKRTDXK6Q, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919502883 (590h34m43.790585928s) [noise=true , address=145.79.227.45:4229 , piece_id=AZBFGM6B4B6XAKIJMEGT3NY542RPI4GYGXQOHRMHN4MBSWIH7HZQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919976906 (590h34m43.791059951s) [noise=true , address=145.79.227.10:4402 , piece_id=RB4ELKAZWOZYJFPVJSSQSQ4A6MQ3QABD64ZOFBMBGBLP4ZBLYTCQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.920435189 (590h34m43.791518234s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.921350631 (590h34m43.792433666s) [noise=true , address=145.79.227.12:4765 , piece_id=IFHTZWQASQV57XXQ4CK4OBDCMCZLFNWA6MK7B3J5SOGDUGKLQ75A, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.922041407 (590h34m43.793124452s) [noise=true , address=45.91.116.5:4297 , piece_id=BC76YOSI7JD6KIC6O2ZIH5EAEQSRUI5TQ33RKEC53PAQA7Q2OLIQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.922881406 (590h34m43.793964461s) [noise=true , address=77.90.31.18:4805 , piece_id=JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.923811317 (590h34m43.794894372s) [noise=true , address=45.91.116.7:4012 , piece_id=ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.924751988 (590h34m43.795835023s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.925483261 (590h34m43.796566306s) [noise=true , address=145.79.227.8:4329 , piece_id=2U24RSR6RO4AUNPUKIL2EHPXFIIJV6J54Q5XV37Z7DUJ47YVLUOA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.926350623 (590h34m43.797433658s) [noise=true , address=145.79.227.11:4731 , piece_id=TIIT6OYWB5WSC2YW44KYDOVM4RLZXFAQDSDVKSRBCBDYJ5XRZRNA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.927963083 (590h34m43.799046118s) [noise=true , address=145.79.227.38:4978 , piece_id=TCCP6WCVDGLV7DRNMZLZFO5FTGLIV2CWEUEOWJ7LW3GJMMASPZMQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.929085090 (590h34m43.800168135s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.930062000 (590h34m43.801145045s) [noise=true , address=145.79.227.5:4223 , piece_id=M52N3HJ37KOPA3LNGSIYM5V57YE4ELOONXUMD54JNIUJLMYGHZHQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:45.617481040 (590h34m45.488564085s) [upload=517732]: Next returned nil segment Dec 30 06:03:45.617511608 (590h34m45.488594654s) [upload=517732]: Waiting for error group managing segments... Dec 30 06:03:45.903336394 (590h34m45.774419439s) [noise=true , address=145.79.227.45:4229 , piece_id=AZBFGM6B4B6XAKIJMEGT3NY542RPI4GYGXQOHRMHN4MBSWIH7HZQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.910012327 (590h34m45.781095372s) [noise=true , address=145.79.227.5:4223 , piece_id=M52N3HJ37KOPA3LNGSIYM5V57YE4ELOONXUMD54JNIUJLMYGHZHQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.914200573 (590h34m45.785283618s) [noise=true , address=145.79.227.10:4402 , piece_id=RB4ELKAZWOZYJFPVJSSQSQ4A6MQ3QABD64ZOFBMBGBLP4ZBLYTCQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.930777117 (590h34m45.801860162s) [noise=true , address=145.79.227.8:4329 , piece_id=2U24RSR6RO4AUNPUKIL2EHPXFIIJV6J54Q5XV37Z7DUJ47YVLUOA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.931789354 (590h34m45.802872399s) [noise=true , address=145.79.227.15:4873 , piece_id=G2V6NLM4Q2JGZOB7CIX224JYFQUXPCQXR6YR3YZ7HU7QNP7LPQRA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.933085292 (590h34m45.804168337s) [noise=true , address=145.79.227.14:4837 , piece_id=2RR2YOJE4Z2HPO662QZMAH4BWUJIVP26T3LUHM4N6Y3CQEPWDB7Q, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.945017462 (590h34m45.816100497s) [noise=true , address=145.79.227.12:4765 , piece_id=IFHTZWQASQV57XXQ4CK4OBDCMCZLFNWA6MK7B3J5SOGDUGKLQ75A, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.997781494 (590h34m45.868864539s) [noise=true , address=145.79.227.11:4731 , piece_id=TIIT6OYWB5WSC2YW44KYDOVM4RLZXFAQDSDVKSRBCBDYJ5XRZRNA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.006849841 (590h34m45.877932876s) [noise=true , address=145.79.227.38:4978 , piece_id=TCCP6WCVDGLV7DRNMZLZFO5FTGLIV2CWEUEOWJ7LW3GJMMASPZMQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.225565652 (590h34m46.096648687s) [noise=true , address=77.90.31.14:4664 , piece_id=V6BZMBCU5Z6ZKUXNZ7E7AGRTKUNTMDEA3532BVP5K2DJKRTDXK6Q, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.265443075 (590h34m46.13652612s) [noise=true , address=45.91.116.5:4297 , piece_id=BC76YOSI7JD6KIC6O2ZIH5EAEQSRUI5TQ33RKEC53PAQA7Q2OLIQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.600316007 (590h34m46.471399052s) [noise=true , address=45.91.116.7:4012 , piece_id=ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.801841444 (590h34m46.672924519s) [noise=true , address=77.90.31.18:4805 , piece_id=JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.801867023 (590h34m46.672950058s) [seg_pos={0 0}, upload=517732]: Segment reached optimal threshold of 13 pieces. Dec 30 06:03:46.802099957 (590h34m46.673183002s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802110587 (590h34m46.673193622s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802131887 (590h34m46.673214932s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802139983 (590h34m46.673223038s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802472696 (590h34m46.673555741s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802483287 (590h34m46.673566322s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802509205 (590h34m46.67359224s) [seg_pos={0 0}, upload=517732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:03:46.802602804 (590h34m46.673685859s) [upload=517732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:03:46.802610218 (590h34m46.673693263s) [upload=517732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:03:47.099426405 (590h34m46.97050946s) [upload=517732]: Done uploading segments... Dec 30 06:03:47.099449608 (590h34m46.970532653s) [upload=517732]: Upload finished. err: <nil> Duration: 19.830013419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 36.65382794s trace: https://jaeger.dev.storj.io/trace/29aa707bf8bd3de6 FAIL (2025-12-30 06:05:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.517379430 (590h34m25.388462475s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.519724064 (590h34m25.39080711s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.522068399 (590h34m25.393151434s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.523362533 (590h34m25.394445578s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.524404988 (590h34m25.395488023s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.525938748 (590h34m25.397021793s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.526464530 (590h34m25.397547575s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.529259672 (590h34m25.400342717s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.531769822 (590h34m25.402852867s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.532925572 (590h34m25.404008607s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.534415720 (590h34m25.405498765s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537220020 (590h34m25.408303065s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537490615 (590h34m25.40857365s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.539400482 (590h34m25.410483517s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.541128883 (590h34m25.412211918s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:26.991260789 (590h34m26.862343834s) [upload=517728]: Next returned nil segment Dec 30 06:03:26.991297679 (590h34m26.862380734s) [upload=517728]: Waiting for error group managing segments... Dec 30 06:03:27.434491232 (590h34m27.305574277s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.473885785 (590h34m27.34496882s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.481026933 (590h34m27.352109968s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.537133078 (590h34m27.408216113s) [noise=true , address=145.79.227.43:4174 , piece_id=PYWCVSCRU45AHKUIGBP6XXOQSR7CS5GIU2W3CV2C5VIAVJAFMYAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.610086884 (590h34m27.481169919s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682091743 (590h34m27.553174788s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682634627 (590h34m27.553717672s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.712364247 (590h34m27.583447292s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.000168182 (590h34m27.871251227s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.289616990 (590h34m28.160700025s) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.306681834 (590h34m28.177764879s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.312426353 (590h34m28.183509388s) [noise=true , address=145.79.227.30:4779 , piece_id=KBETI2JQ3NDRDFTGFA5FW7S5IYL2TFIZ3NXTKQZX7JZQ5RTMX2LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.349481942 (590h34m28.220564987s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380570801 (590h34m28.251653836s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380591970 (590h34m28.251675016s) [seg_pos={0 0}, upload=517728]: Segment reached optimal threshold of 14 pieces. Dec 30 06:03:28.380822891 (590h34m28.251905926s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380830565 (590h34m28.2519136s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380925726 (590h34m28.252008771s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection Dec 30 06:03:28.380942197 (590h34m28.252025242s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380958769 (590h34m28.252041814s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380967896 (590h34m28.252050942s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381091813 (590h34m28.252174858s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.381098525 (590h34m28.252181561s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381144553 (590h34m28.252227588s) [seg_pos={0 0}, upload=517728]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:03:28.381225898 (590h34m28.252308933s) [upload=517728]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:03:28.381231308 (590h34m28.252314344s) [upload=517728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:03:28.673388913 (590h34m28.544471958s) [upload=517728]: Done uploading segments... Dec 30 06:03:28.673412569 (590h34m28.544495604s) [upload=517728]: Upload finished. err: <nil> Duration: 36.653850211s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.96941717s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:04:47 UTC) orbiter warning: context deadline exceeded Duration: 3.969446636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.004412389s trace: https://jaeger.dev.storj.io/trace/29aa707bf8bd3de6 FAIL (2025-12-30 06:04:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.517379430 (590h34m25.388462475s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.519724064 (590h34m25.39080711s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.522068399 (590h34m25.393151434s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.523362533 (590h34m25.394445578s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.524404988 (590h34m25.395488023s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.525938748 (590h34m25.397021793s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.526464530 (590h34m25.397547575s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.529259672 (590h34m25.400342717s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.531769822 (590h34m25.402852867s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.532925572 (590h34m25.404008607s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.534415720 (590h34m25.405498765s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537220020 (590h34m25.408303065s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537490615 (590h34m25.40857365s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.539400482 (590h34m25.410483517s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.541128883 (590h34m25.412211918s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:26.991260789 (590h34m26.862343834s) [upload=517728]: Next returned nil segment Dec 30 06:03:26.991297679 (590h34m26.862380734s) [upload=517728]: Waiting for error group managing segments... Dec 30 06:03:27.434491232 (590h34m27.305574277s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.473885785 (590h34m27.34496882s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.481026933 (590h34m27.352109968s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.537133078 (590h34m27.408216113s) [noise=true , address=145.79.227.43:4174 , piece_id=PYWCVSCRU45AHKUIGBP6XXOQSR7CS5GIU2W3CV2C5VIAVJAFMYAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.610086884 (590h34m27.481169919s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682091743 (590h34m27.553174788s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682634627 (590h34m27.553717672s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.712364247 (590h34m27.583447292s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.000168182 (590h34m27.871251227s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.289616990 (590h34m28.160700025s) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.306681834 (590h34m28.177764879s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.312426353 (590h34m28.183509388s) [noise=true , address=145.79.227.30:4779 , piece_id=KBETI2JQ3NDRDFTGFA5FW7S5IYL2TFIZ3NXTKQZX7JZQ5RTMX2LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.349481942 (590h34m28.220564987s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380570801 (590h34m28.251653836s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380591970 (590h34m28.251675016s) [seg_pos={0 0}, upload=517728]: Segment reached optimal threshold of 14 pieces. Dec 30 06:03:28.380822891 (590h34m28.251905926s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380830565 (590h34m28.2519136s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380925726 (590h34m28.252008771s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection Dec 30 06:03:28.380942197 (590h34m28.252025242s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380958769 (590h34m28.252041814s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380967896 (590h34m28.252050942s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381091813 (590h34m28.252174858s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.381098525 (590h34m28.252181561s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381144553 (590h34m28.252227588s) [seg_pos={0 0}, upload=517728]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:03:28.381225898 (590h34m28.252308933s) [upload=517728]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:03:28.381231308 (590h34m28.252314344s) [upload=517728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:03:28.673388913 (590h34m28.544471958s) [upload=517728]: Done uploading segments... Dec 30 06:03:28.673412569 (590h34m28.544495604s) [upload=517728]: Upload finished. err: <nil> Duration: 1m0.004427459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......X.] duration: 5.768885468s trace: https://jaeger.dev.storj.io/trace/1237689ed7bb48cb FAIL (2025-12-30 06:04:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA) context deadline exceeded ======LOGS====== Dec 30 06:03:43.669744944 (590h34m43.540827989s) [upload=517732]: Starting upload Dec 30 06:03:43.669856857 (590h34m43.540939902s) [upload=517732]: Done starting upload Dec 30 06:03:43.670047460 (590h34m43.541130515s) [upload=517732]: Uploading segments... Dec 30 06:03:43.670118325 (590h34m43.54120138s) [upload=517732]: Got next segment. Inline: false Dec 30 06:03:43.670156988 (590h34m43.541240063s) [upload=517732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:03:43.670163641 (590h34m43.541246686s) [upload=517732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:03:43.917142178 (590h34m43.788225213s) [seg_pos={0 0}, upload=517732]: Begin upload segment... Dec 30 06:03:43.917189789 (590h34m43.788272834s) [seg_pos={0 0}, upload=517732]: Done begin upload segment. Dec 30 06:03:43.917215047 (590h34m43.788298082s) [noise=true , address=145.79.227.14:4837 , piece_id=2RR2YOJE4Z2HPO662QZMAH4BWUJIVP26T3LUHM4N6Y3CQEPWDB7Q, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.918021964 (590h34m43.789105019s) [noise=true , address=145.79.227.15:4873 , piece_id=G2V6NLM4Q2JGZOB7CIX224JYFQUXPCQXR6YR3YZ7HU7QNP7LPQRA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919021606 (590h34m43.790104651s) [noise=true , address=77.90.31.14:4664 , piece_id=V6BZMBCU5Z6ZKUXNZ7E7AGRTKUNTMDEA3532BVP5K2DJKRTDXK6Q, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919502883 (590h34m43.790585928s) [noise=true , address=145.79.227.45:4229 , piece_id=AZBFGM6B4B6XAKIJMEGT3NY542RPI4GYGXQOHRMHN4MBSWIH7HZQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919976906 (590h34m43.791059951s) [noise=true , address=145.79.227.10:4402 , piece_id=RB4ELKAZWOZYJFPVJSSQSQ4A6MQ3QABD64ZOFBMBGBLP4ZBLYTCQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.920435189 (590h34m43.791518234s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.921350631 (590h34m43.792433666s) [noise=true , address=145.79.227.12:4765 , piece_id=IFHTZWQASQV57XXQ4CK4OBDCMCZLFNWA6MK7B3J5SOGDUGKLQ75A, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.922041407 (590h34m43.793124452s) [noise=true , address=45.91.116.5:4297 , piece_id=BC76YOSI7JD6KIC6O2ZIH5EAEQSRUI5TQ33RKEC53PAQA7Q2OLIQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.922881406 (590h34m43.793964461s) [noise=true , address=77.90.31.18:4805 , piece_id=JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.923811317 (590h34m43.794894372s) [noise=true , address=45.91.116.7:4012 , piece_id=ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.924751988 (590h34m43.795835023s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.925483261 (590h34m43.796566306s) [noise=true , address=145.79.227.8:4329 , piece_id=2U24RSR6RO4AUNPUKIL2EHPXFIIJV6J54Q5XV37Z7DUJ47YVLUOA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.926350623 (590h34m43.797433658s) [noise=true , address=145.79.227.11:4731 , piece_id=TIIT6OYWB5WSC2YW44KYDOVM4RLZXFAQDSDVKSRBCBDYJ5XRZRNA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.927963083 (590h34m43.799046118s) [noise=true , address=145.79.227.38:4978 , piece_id=TCCP6WCVDGLV7DRNMZLZFO5FTGLIV2CWEUEOWJ7LW3GJMMASPZMQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.929085090 (590h34m43.800168135s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.930062000 (590h34m43.801145045s) [noise=true , address=145.79.227.5:4223 , piece_id=M52N3HJ37KOPA3LNGSIYM5V57YE4ELOONXUMD54JNIUJLMYGHZHQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:45.617481040 (590h34m45.488564085s) [upload=517732]: Next returned nil segment Dec 30 06:03:45.617511608 (590h34m45.488594654s) [upload=517732]: Waiting for error group managing segments... Dec 30 06:03:45.903336394 (590h34m45.774419439s) [noise=true , address=145.79.227.45:4229 , piece_id=AZBFGM6B4B6XAKIJMEGT3NY542RPI4GYGXQOHRMHN4MBSWIH7HZQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.910012327 (590h34m45.781095372s) [noise=true , address=145.79.227.5:4223 , piece_id=M52N3HJ37KOPA3LNGSIYM5V57YE4ELOONXUMD54JNIUJLMYGHZHQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.914200573 (590h34m45.785283618s) [noise=true , address=145.79.227.10:4402 , piece_id=RB4ELKAZWOZYJFPVJSSQSQ4A6MQ3QABD64ZOFBMBGBLP4ZBLYTCQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.930777117 (590h34m45.801860162s) [noise=true , address=145.79.227.8:4329 , piece_id=2U24RSR6RO4AUNPUKIL2EHPXFIIJV6J54Q5XV37Z7DUJ47YVLUOA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.931789354 (590h34m45.802872399s) [noise=true , address=145.79.227.15:4873 , piece_id=G2V6NLM4Q2JGZOB7CIX224JYFQUXPCQXR6YR3YZ7HU7QNP7LPQRA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.933085292 (590h34m45.804168337s) [noise=true , address=145.79.227.14:4837 , piece_id=2RR2YOJE4Z2HPO662QZMAH4BWUJIVP26T3LUHM4N6Y3CQEPWDB7Q, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.945017462 (590h34m45.816100497s) [noise=true , address=145.79.227.12:4765 , piece_id=IFHTZWQASQV57XXQ4CK4OBDCMCZLFNWA6MK7B3J5SOGDUGKLQ75A, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.997781494 (590h34m45.868864539s) [noise=true , address=145.79.227.11:4731 , piece_id=TIIT6OYWB5WSC2YW44KYDOVM4RLZXFAQDSDVKSRBCBDYJ5XRZRNA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.006849841 (590h34m45.877932876s) [noise=true , address=145.79.227.38:4978 , piece_id=TCCP6WCVDGLV7DRNMZLZFO5FTGLIV2CWEUEOWJ7LW3GJMMASPZMQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.225565652 (590h34m46.096648687s) [noise=true , address=77.90.31.14:4664 , piece_id=V6BZMBCU5Z6ZKUXNZ7E7AGRTKUNTMDEA3532BVP5K2DJKRTDXK6Q, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.265443075 (590h34m46.13652612s) [noise=true , address=45.91.116.5:4297 , piece_id=BC76YOSI7JD6KIC6O2ZIH5EAEQSRUI5TQ33RKEC53PAQA7Q2OLIQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.600316007 (590h34m46.471399052s) [noise=true , address=45.91.116.7:4012 , piece_id=ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.801841444 (590h34m46.672924519s) [noise=true , address=77.90.31.18:4805 , piece_id=JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.801867023 (590h34m46.672950058s) [seg_pos={0 0}, upload=517732]: Segment reached optimal threshold of 13 pieces. Dec 30 06:03:46.802099957 (590h34m46.673183002s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802110587 (590h34m46.673193622s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802131887 (590h34m46.673214932s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802139983 (590h34m46.673223038s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802472696 (590h34m46.673555741s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802483287 (590h34m46.673566322s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802509205 (590h34m46.67359224s) [seg_pos={0 0}, upload=517732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:03:46.802602804 (590h34m46.673685859s) [upload=517732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:03:46.802610218 (590h34m46.673693263s) [upload=517732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:03:47.099426405 (590h34m46.97050946s) [upload=517732]: Done uploading segments... Dec 30 06:03:47.099449608 (590h34m46.970532653s) [upload=517732]: Upload finished. err: <nil> Duration: 5.76890227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......X] duration: 30.003401378s trace: https://jaeger.dev.storj.io/trace/1237689ed7bb48cb FAIL (2025-12-30 06:04:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA) context deadline exceeded ======LOGS====== Dec 30 06:03:43.669744944 (590h34m43.540827989s) [upload=517732]: Starting upload Dec 30 06:03:43.669856857 (590h34m43.540939902s) [upload=517732]: Done starting upload Dec 30 06:03:43.670047460 (590h34m43.541130515s) [upload=517732]: Uploading segments... Dec 30 06:03:43.670118325 (590h34m43.54120138s) [upload=517732]: Got next segment. Inline: false Dec 30 06:03:43.670156988 (590h34m43.541240063s) [upload=517732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:03:43.670163641 (590h34m43.541246686s) [upload=517732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:03:43.917142178 (590h34m43.788225213s) [seg_pos={0 0}, upload=517732]: Begin upload segment... Dec 30 06:03:43.917189789 (590h34m43.788272834s) [seg_pos={0 0}, upload=517732]: Done begin upload segment. Dec 30 06:03:43.917215047 (590h34m43.788298082s) [noise=true , address=145.79.227.14:4837 , piece_id=2RR2YOJE4Z2HPO662QZMAH4BWUJIVP26T3LUHM4N6Y3CQEPWDB7Q, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.918021964 (590h34m43.789105019s) [noise=true , address=145.79.227.15:4873 , piece_id=G2V6NLM4Q2JGZOB7CIX224JYFQUXPCQXR6YR3YZ7HU7QNP7LPQRA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919021606 (590h34m43.790104651s) [noise=true , address=77.90.31.14:4664 , piece_id=V6BZMBCU5Z6ZKUXNZ7E7AGRTKUNTMDEA3532BVP5K2DJKRTDXK6Q, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919502883 (590h34m43.790585928s) [noise=true , address=145.79.227.45:4229 , piece_id=AZBFGM6B4B6XAKIJMEGT3NY542RPI4GYGXQOHRMHN4MBSWIH7HZQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919976906 (590h34m43.791059951s) [noise=true , address=145.79.227.10:4402 , piece_id=RB4ELKAZWOZYJFPVJSSQSQ4A6MQ3QABD64ZOFBMBGBLP4ZBLYTCQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.920435189 (590h34m43.791518234s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.921350631 (590h34m43.792433666s) [noise=true , address=145.79.227.12:4765 , piece_id=IFHTZWQASQV57XXQ4CK4OBDCMCZLFNWA6MK7B3J5SOGDUGKLQ75A, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.922041407 (590h34m43.793124452s) [noise=true , address=45.91.116.5:4297 , piece_id=BC76YOSI7JD6KIC6O2ZIH5EAEQSRUI5TQ33RKEC53PAQA7Q2OLIQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.922881406 (590h34m43.793964461s) [noise=true , address=77.90.31.18:4805 , piece_id=JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.923811317 (590h34m43.794894372s) [noise=true , address=45.91.116.7:4012 , piece_id=ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.924751988 (590h34m43.795835023s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.925483261 (590h34m43.796566306s) [noise=true , address=145.79.227.8:4329 , piece_id=2U24RSR6RO4AUNPUKIL2EHPXFIIJV6J54Q5XV37Z7DUJ47YVLUOA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.926350623 (590h34m43.797433658s) [noise=true , address=145.79.227.11:4731 , piece_id=TIIT6OYWB5WSC2YW44KYDOVM4RLZXFAQDSDVKSRBCBDYJ5XRZRNA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.927963083 (590h34m43.799046118s) [noise=true , address=145.79.227.38:4978 , piece_id=TCCP6WCVDGLV7DRNMZLZFO5FTGLIV2CWEUEOWJ7LW3GJMMASPZMQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.929085090 (590h34m43.800168135s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.930062000 (590h34m43.801145045s) [noise=true , address=145.79.227.5:4223 , piece_id=M52N3HJ37KOPA3LNGSIYM5V57YE4ELOONXUMD54JNIUJLMYGHZHQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:45.617481040 (590h34m45.488564085s) [upload=517732]: Next returned nil segment Dec 30 06:03:45.617511608 (590h34m45.488594654s) [upload=517732]: Waiting for error group managing segments... Dec 30 06:03:45.903336394 (590h34m45.774419439s) [noise=true , address=145.79.227.45:4229 , piece_id=AZBFGM6B4B6XAKIJMEGT3NY542RPI4GYGXQOHRMHN4MBSWIH7HZQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.910012327 (590h34m45.781095372s) [noise=true , address=145.79.227.5:4223 , piece_id=M52N3HJ37KOPA3LNGSIYM5V57YE4ELOONXUMD54JNIUJLMYGHZHQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.914200573 (590h34m45.785283618s) [noise=true , address=145.79.227.10:4402 , piece_id=RB4ELKAZWOZYJFPVJSSQSQ4A6MQ3QABD64ZOFBMBGBLP4ZBLYTCQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.930777117 (590h34m45.801860162s) [noise=true , address=145.79.227.8:4329 , piece_id=2U24RSR6RO4AUNPUKIL2EHPXFIIJV6J54Q5XV37Z7DUJ47YVLUOA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.931789354 (590h34m45.802872399s) [noise=true , address=145.79.227.15:4873 , piece_id=G2V6NLM4Q2JGZOB7CIX224JYFQUXPCQXR6YR3YZ7HU7QNP7LPQRA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.933085292 (590h34m45.804168337s) [noise=true , address=145.79.227.14:4837 , piece_id=2RR2YOJE4Z2HPO662QZMAH4BWUJIVP26T3LUHM4N6Y3CQEPWDB7Q, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.945017462 (590h34m45.816100497s) [noise=true , address=145.79.227.12:4765 , piece_id=IFHTZWQASQV57XXQ4CK4OBDCMCZLFNWA6MK7B3J5SOGDUGKLQ75A, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.997781494 (590h34m45.868864539s) [noise=true , address=145.79.227.11:4731 , piece_id=TIIT6OYWB5WSC2YW44KYDOVM4RLZXFAQDSDVKSRBCBDYJ5XRZRNA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.006849841 (590h34m45.877932876s) [noise=true , address=145.79.227.38:4978 , piece_id=TCCP6WCVDGLV7DRNMZLZFO5FTGLIV2CWEUEOWJ7LW3GJMMASPZMQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.225565652 (590h34m46.096648687s) [noise=true , address=77.90.31.14:4664 , piece_id=V6BZMBCU5Z6ZKUXNZ7E7AGRTKUNTMDEA3532BVP5K2DJKRTDXK6Q, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.265443075 (590h34m46.13652612s) [noise=true , address=45.91.116.5:4297 , piece_id=BC76YOSI7JD6KIC6O2ZIH5EAEQSRUI5TQ33RKEC53PAQA7Q2OLIQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.600316007 (590h34m46.471399052s) [noise=true , address=45.91.116.7:4012 , piece_id=ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.801841444 (590h34m46.672924519s) [noise=true , address=77.90.31.18:4805 , piece_id=JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.801867023 (590h34m46.672950058s) [seg_pos={0 0}, upload=517732]: Segment reached optimal threshold of 13 pieces. Dec 30 06:03:46.802099957 (590h34m46.673183002s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802110587 (590h34m46.673193622s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802131887 (590h34m46.673214932s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802139983 (590h34m46.673223038s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802472696 (590h34m46.673555741s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802483287 (590h34m46.673566322s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802509205 (590h34m46.67359224s) [seg_pos={0 0}, upload=517732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:03:46.802602804 (590h34m46.673685859s) [upload=517732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:03:46.802610218 (590h34m46.673693263s) [upload=517732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:03:47.099426405 (590h34m46.97050946s) [upload=517732]: Done uploading segments... Dec 30 06:03:47.099449608 (590h34m46.970532653s) [upload=517732]: Upload finished. err: <nil> Duration: 30.003419032s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.717333804s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:04:06 UTC) orbiter warning: context deadline exceeded Duration: 3.717363141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......] duration: 6.075970322s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 06:03:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 6.076009457s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.413091029s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:03:13 UTC) orbiter warning: context deadline exceeded Duration: 3.413115534s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001168582s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:02:56 UTC) orbiter warning: context deadline exceeded Duration: 5.001192788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX......] duration: 5.945563877s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 06:02:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 5.945588264s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 4.66313794s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 06:01:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.663168547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.....] duration: 12.221682745s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 06:01:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 12.221708213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX....] duration: 5.797011388s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 06:01:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 5.797045032s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.423401851s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 06:00:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.423582826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX...] duration: 6.138223834s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 06:00:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 6.138283118s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.650910745s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 06:00:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.650932436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX..] duration: 6.020037076s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 05:59:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 6.020064268s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 4.895656702s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:59:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.895685188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.430983828s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:59:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.43102173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.] duration: 5.817613058s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 05:58:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 5.817643786s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)