Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X........X...] duration: 34.694491182s trace: https://jaeger.dev.storj.io/trace/29b498816f392fc6 FAIL (2025-12-28 17:54:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6770948 (554h22m7.427853983s) [noise=true , address=77.90.31.13:4637 , piece_id=7IQITGZPXZVGGJE5WJUTIUPJQG7G6R2UXLQIEIDCSZPEFSZGO7GA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.556876499 (554h22m7.427959544s) [noise=true , address=145.79.227.44:4208 , piece_id=MX6FZVSTYGJCJ5DD3LXV36FRXMRTDY5VHPFZEKBNDYVNBEBSEPGQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557122898 (554h22m7.428205943s) [noise=true , address=77.90.31.16:4767 , piece_id=2IMQX3JLVFYYCQ6LKBLDVNDB6X3MMSTFG2YN6NZRCIR2QOS53HUQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557323139 (554h22m7.428406184s) [noise=true , address=145.79.227.25:4624 , piece_id=O57LSKM4YSSIA7N5K7VVP6XO6WQ4VQGZXVRQTBIRTVQP462SV65Q, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557456964 (554h22m7.428540009s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557722129 (554h22m7.428805165s) [noise=true , address=145.79.227.36:4549 , piece_id=TO6PZ3W33Q5EYTEEWOPYT52PFIRVJCNUTV2PNMWUGLHFBMTCJBGQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557973388 (554h22m7.429056423s) [noise=true , address=145.79.227.33:4882 , piece_id=TELMFJG4H5HSTI545736O452TWVRZ4OVVU54B3CEITUXWPRAOLHA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558373519 (554h22m7.429456564s) [noise=true , address=145.79.227.49:4342 , piece_id=X3LJ7ACO2LQTJBIGEQPIM2LY55ILNW6THFE6BWCZYGA2UKKKVFFQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558388057 (554h22m7.429471092s) [noise=true , address=145.79.227.29:4969 , piece_id=SOEYSHES5BYEGPBQAS24MCKRHH4ALL5FVEDUKKFIVWX3OAOD5Y6A, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558791585 (554h22m7.42987463s) [noise=true , address=145.79.227.31:4809 , piece_id=WYJ2EU3QN3FFZFL5QTP5CNAS5BCIUIFMH63C54ZLQJZY54TPRLVA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558823486 (554h22m7.429906531s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559230000 (554h22m7.430313035s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559247845 (554h22m7.43033089s) [noise=true , address=145.79.227.21:4481 , piece_id=MEAY63J5KAYJJ4TJP2IGY2PC6NJEOVRZ24QY6TQD4ATGBURA5QCQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559656883 (554h22m7.430739918s) [noise=true , address=193.149.164.143:4702 , piece_id=RERLF6JJJ7DWQVYJDSDET5OOWHADUSJXWL2VJJI56JINOZIGDHWA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.562207000 (554h22m7.433290055s) [noise=true , address=64.49.13.140:4516 , piece_id=72KQD7KPLSQUNDS3ILVTZTSXGQ6UF7MD3P2CUJ5CL7IAC2MFSI7A, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.564270158 (554h22m7.435353313s) [noise=true , address=45.91.116.5:4312 , piece_id=ZNPVKTONLL6NFMHUNS53ERZCRIJMA3XISFSS2NBEZOKKWQZO6ECQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:08.488386820 (554h22m8.359469855s) [noise=true , address=193.149.164.143:4702 , piece_id=RERLF6JJJ7DWQVYJDSDET5OOWHADUSJXWL2VJJI56JINOZIGDHWA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 28 17:51:08.868542955 (554h22m8.73962599s) [upload=486027]: Next returned nil segment Dec 28 17:51:08.868558364 (554h22m8.739641409s) [upload=486027]: Waiting for error group managing segments... Dec 28 17:51:09.298510128 (554h22m9.169593163s) [noise=true , address=64.49.13.137:4503 , piece_id=WZZ36TJ4V76JHORSPNBADMDSL6YYO3EW4DEQKDK3VIMZ4F3G7JSQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.396193335 (554h22m9.26727639s) [noise=true , address=64.49.13.140:4516 , piece_id=72KQD7KPLSQUNDS3ILVTZTSXGQ6UF7MD3P2CUJ5CL7IAC2MFSI7A, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.426697790 (554h22m9.297780835s) [noise=true , address=145.79.227.36:4549 , piece_id=TO6PZ3W33Q5EYTEEWOPYT52PFIRVJCNUTV2PNMWUGLHFBMTCJBGQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.441962316 (554h22m9.313045361s) [noise=true , address=145.79.227.31:4809 , piece_id=WYJ2EU3QN3FFZFL5QTP5CNAS5BCIUIFMH63C54ZLQJZY54TPRLVA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.443532056 (554h22m9.314615101s) [noise=true , address=145.79.227.25:4624 , piece_id=O57LSKM4YSSIA7N5K7VVP6XO6WQ4VQGZXVRQTBIRTVQP462SV65Q, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.448848529 (554h22m9.319931574s) [noise=true , address=145.79.227.21:4481 , piece_id=MEAY63J5KAYJJ4TJP2IGY2PC6NJEOVRZ24QY6TQD4ATGBURA5QCQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.449293015 (554h22m9.32037605s) [noise=true , address=145.79.227.29:4969 , piece_id=SOEYSHES5BYEGPBQAS24MCKRHH4ALL5FVEDUKKFIVWX3OAOD5Y6A, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.520315725 (554h22m9.391398771s) [noise=true , address=145.79.227.19:4100 , piece_id=X4NAGMU2TFRKKTTETO7AV2LJO2BPDI6TPH32NQFWGNYUAXN3UDRQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.645133701 (554h22m9.516216746s) [noise=true , address=77.90.31.13:4637 , piece_id=7IQITGZPXZVGGJE5WJUTIUPJQG7G6R2UXLQIEIDCSZPEFSZGO7GA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.670952274 (554h22m9.542035309s) [noise=true , address=45.91.116.5:4312 , piece_id=ZNPVKTONLL6NFMHUNS53ERZCRIJMA3XISFSS2NBEZOKKWQZO6ECQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.693198264 (554h22m9.564281309s) [noise=true , address=77.90.31.16:4767 , piece_id=2IMQX3JLVFYYCQ6LKBLDVNDB6X3MMSTFG2YN6NZRCIR2QOS53HUQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.298273265 (554h22m10.16935631s) [noise=true , address=145.79.227.33:4882 , piece_id=TELMFJG4H5HSTI545736O452TWVRZ4OVVU54B3CEITUXWPRAOLHA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.377531416 (554h22m10.248614451s) [noise=true , address=145.79.227.49:4342 , piece_id=X3LJ7ACO2LQTJBIGEQPIM2LY55ILNW6THFE6BWCZYGA2UKKKVFFQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.382353338 (554h22m10.253436373s) [noise=true , address=145.79.227.44:4208 , piece_id=MX6FZVSTYGJCJ5DD3LXV36FRXMRTDY5VHPFZEKBNDYVNBEBSEPGQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.382374157 (554h22m10.253457202s) [seg_pos={0 0}, upload=486027]: Segment reached optimal threshold of 14 pieces. Dec 28 17:51:10.382598706 (554h22m10.253681751s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection; write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection Dec 28 17:51:10.382606851 (554h22m10.253689896s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382812302 (554h22m10.253895337s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection; write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection Dec 28 17:51:10.382819997 (554h22m10.253903032s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382942811 (554h22m10.254025846s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 28 17:51:10.382948151 (554h22m10.254031186s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382963951 (554h22m10.254046996s) [seg_pos={0 0}, upload=486027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:51:10.383031409 (554h22m10.254114454s) [upload=486027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:51:10.383035807 (554h22m10.254118852s) [upload=486027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:51:10.688025582 (554h22m10.559108617s) [upload=486027]: Done uploading segments... Dec 28 17:51:10.688114761 (554h22m10.559197796s) [upload=486027]: Upload finished. err: <nil> Duration: 34.694507934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........X..XX.X.XX.] duration: 5.695430951s trace: https://jaeger.dev.storj.io/trace/21f865b8a1414caa FAIL (2025-12-28 17:53:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ments... Dec 28 17:52:32.803342801 (554h23m32.674425836s) [upload=486048]: Got next segment. Inline: false Dec 28 17:52:32.803387345 (554h23m32.67447039s) [upload=486048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:52:32.803392775 (554h23m32.674475811s) [upload=486048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:52:33.045334645 (554h23m32.91641768s) [seg_pos={0 0}, upload=486048]: Begin upload segment... Dec 28 17:52:33.045392225 (554h23m32.91647526s) [seg_pos={0 0}, upload=486048]: Done begin upload segment. Dec 28 17:52:33.045418886 (554h23m32.916501921s) [noise=true , address=145.79.227.51:4410 , piece_id=CWQ3CLCDYYSAKX5LJ74ZPNBN6DNUOEGRK54ZWZREK2PFGQQ45J4A, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.045540017 (554h23m32.916623052s) [noise=true , address=145.79.227.16:4909 , piece_id=37PO6YVMUFLFW4W5WPFPLBMB6PMFHURXMR77A65QWZUEDIVTMJQQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.046097789 (554h23m32.917180824s) [noise=true , address=45.91.116.6:4976 , piece_id=7QMUCDH5PKQ7D4U6CXTZMZ4HWJPZH5WZ7AXX5FJKCCGM5IBD7SMA, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.046386147 (554h23m32.917469182s) [noise=true , address=145.79.227.13:4803 , piece_id=FQX4NPB2ROF2YQK7QN7PYYCUBDMRJRN6H37IHJXUU3O6JMTRY5NQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.046622327 (554h23m32.917705372s) [noise=true , address=145.79.227.50:4375 , piece_id=HBXUI7G4HUDSS6SOOJHIYQDDKPAPF3A2Y7JBXBMQ2YKGQ32V4CXQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.046677433 (554h23m32.917760468s) [noise=true , address=77.90.31.13:4629 , piece_id=TD462EOSCOASQSBLJMBEGWJQMMZ2EOEOJCOUISL75JVE452VIKLQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.046986792 (554h23m32.918069837s) [noise=true , address=77.90.31.19:4841 , piece_id=WJKDMHYCOYBNH77VSRL5TECRGBV35X3WKVVKS7ANBF5KIBX6V5NA, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.047322410 (554h23m32.918405455s) [noise=true , address=145.79.227.47:4267 , piece_id=GA3NJPN3TUS2C5VKYMHXPER77JTMBDFECBYXGU3JOST6KYULMBSQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.047347208 (554h23m32.918430253s) [noise=true , address=145.79.227.52:4447 , piece_id=IYIN2CIIO3LH3N32DDKDDNO4NCAYHKZV6JHAFOOBVVKIAXBEHKLQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.047663209 (554h23m32.918746245s) [noise=true , address=145.79.227.22:4510 , piece_id=UORFHYUBO3MOR4XJMQLDMW3CJIH4BGNAG5OTYRJLQU7NPMUZZ7EQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.047928465 (554h23m32.91901151s) [noise=true , address=45.91.116.14:4192 , piece_id=2WO7WIZTI5VUXNU5ZN6Y5OM7IQ4BQAB4A3LMUFBPF4DGIPCTYFUQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.048571349 (554h23m32.919654394s) [noise=true , address=145.79.227.2:4114 , piece_id=ZK4IOQNIWABK5U5UJJXUJUZFNFUHGCX2TXJTS2DIHQ7AXEBZ3FEQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.049213672 (554h23m32.920296717s) [noise=true , address=45.91.116.10:4081 , piece_id=5WSM23ARPD2LSJCL5DXDZASUYNXXOJZW5G2BH6JDRXXTYVBMVHCQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.049942821 (554h23m32.921025856s) [noise=true , address=145.79.227.45:4231 , piece_id=WDPMNFMIVW554T6KWK4A3Z2OAUDBU6UKLETYC3YGJKJGFDRNZDEA, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.050606705 (554h23m32.92168975s) [noise=true , address=145.79.227.26:4653 , piece_id=P7OBLIBIUDIM5CJYABXIDA2HWWCVR7UCHZF5CS5SCRX3OHBNCCKA, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.051595838 (554h23m32.922678883s) [noise=true , address=45.91.116.5:4298 , piece_id=PAR6V2V2GLCMEVXSIVNSRGJFV4VXYBTOA2MVAZWI5RUNSATHQOOA, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:34.760797962 (554h23m34.631880997s) [upload=486048]: Next returned nil segment Dec 28 17:52:34.760813341 (554h23m34.631896386s) [upload=486048]: Waiting for error group managing segments... Dec 28 17:52:35.046758622 (554h23m34.917841657s) [noise=true , address=145.79.227.22:4510 , piece_id=UORFHYUBO3MOR4XJMQLDMW3CJIH4BGNAG5OTYRJLQU7NPMUZZ7EQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.050191930 (554h23m34.921274975s) [noise=true , address=145.79.227.51:4410 , piece_id=CWQ3CLCDYYSAKX5LJ74ZPNBN6DNUOEGRK54ZWZREK2PFGQQ45J4A, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.056530950 (554h23m34.927613985s) [noise=true , address=145.79.227.50:4375 , piece_id=HBXUI7G4HUDSS6SOOJHIYQDDKPAPF3A2Y7JBXBMQ2YKGQ32V4CXQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.065906543 (554h23m34.936989578s) [noise=true , address=145.79.227.2:4114 , piece_id=ZK4IOQNIWABK5U5UJJXUJUZFNFUHGCX2TXJTS2DIHQ7AXEBZ3FEQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.066317215 (554h23m34.93740027s) [noise=true , address=145.79.227.13:4803 , piece_id=FQX4NPB2ROF2YQK7QN7PYYCUBDMRJRN6H37IHJXUU3O6JMTRY5NQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.070497685 (554h23m34.94158072s) [noise=true , address=145.79.227.45:4231 , piece_id=WDPMNFMIVW554T6KWK4A3Z2OAUDBU6UKLETYC3YGJKJGFDRNZDEA, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.071050718 (554h23m34.942133753s) [noise=true , address=145.79.227.52:4447 , piece_id=IYIN2CIIO3LH3N32DDKDDNO4NCAYHKZV6JHAFOOBVVKIAXBEHKLQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.083315753 (554h23m34.954398788s) [noise=true , address=145.79.227.16:4909 , piece_id=37PO6YVMUFLFW4W5WPFPLBMB6PMFHURXMR77A65QWZUEDIVTMJQQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.100352975 (554h23m34.97143602s) [noise=true , address=145.79.227.47:4267 , piece_id=GA3NJPN3TUS2C5VKYMHXPER77JTMBDFECBYXGU3JOST6KYULMBSQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.306619943 (554h23m35.177702979s) [noise=true , address=77.90.31.19:4841 , piece_id=WJKDMHYCOYBNH77VSRL5TECRGBV35X3WKVVKS7ANBF5KIBX6V5NA, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.312275022 (554h23m35.183358057s) [noise=true , address=45.91.116.14:4192 , piece_id=2WO7WIZTI5VUXNU5ZN6Y5OM7IQ4BQAB4A3LMUFBPF4DGIPCTYFUQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.390502781 (554h23m35.261585826s) [noise=true , address=45.91.116.10:4081 , piece_id=5WSM23ARPD2LSJCL5DXDZASUYNXXOJZW5G2BH6JDRXXTYVBMVHCQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.431665262 (554h23m35.302748307s) [noise=true , address=45.91.116.5:4298 , piece_id=PAR6V2V2GLCMEVXSIVNSRGJFV4VXYBTOA2MVAZWI5RUNSATHQOOA, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.431691231 (554h23m35.302774276s) [seg_pos={0 0}, upload=486048]: Segment reached optimal threshold of 13 pieces. Dec 28 17:52:35.431926018 (554h23m35.303009063s) [noise=true , address=145.79.227.26:4653 , piece_id=P7OBLIBIUDIM5CJYABXIDA2HWWCVR7UCHZF5CS5SCRX3OHBNCCKA, seg_pos={0 0}, upload=486048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:45356->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:45356->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45356->145.79.227.26:4653: use of closed network connection Dec 28 17:52:35.431935205 (554h23m35.303018241s) [noise=true , address=145.79.227.26:4653 , piece_id=P7OBLIBIUDIM5CJYABXIDA2HWWCVR7UCHZF5CS5SCRX3OHBNCCKA, seg_pos={0 0}, upload=486048]: Long tail cancellation due to optimal threshold Dec 28 17:52:35.432249043 (554h23m35.303332088s) [noise=true , address=77.90.31.13:4629 , piece_id=TD462EOSCOASQSBLJMBEGWJQMMZ2EOEOJCOUISL75JVE452VIKLQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:34960->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:34960->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34960->77.90.31.13:4629: use of closed network connection Dec 28 17:52:35.432258261 (554h23m35.303341296s) [noise=true , address=77.90.31.13:4629 , piece_id=TD462EOSCOASQSBLJMBEGWJQMMZ2EOEOJCOUISL75JVE452VIKLQ, seg_pos={0 0}, upload=486048]: Long tail cancellation due to optimal threshold Dec 28 17:52:35.432495253 (554h23m35.303578298s) [noise=true , address=45.91.116.6:4976 , piece_id=7QMUCDH5PKQ7D4U6CXTZMZ4HWJPZH5WZ7AXX5FJKCCGM5IBD7SMA, seg_pos={0 0}, upload=486048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 17:52:35.432548824 (554h23m35.303631889s) [noise=true , address=45.91.116.6:4976 , piece_id=7QMUCDH5PKQ7D4U6CXTZMZ4HWJPZH5WZ7AXX5FJKCCGM5IBD7SMA, seg_pos={0 0}, upload=486048]: Long tail cancellation due to optimal threshold Dec 28 17:52:35.432574994 (554h23m35.303658039s) [seg_pos={0 0}, upload=486048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:52:35.432658924 (554h23m35.303741969s) [upload=486048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:52:35.432665676 (554h23m35.303748722s) [upload=486048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:52:35.722617406 (554h23m35.593700451s) [upload=486048]: Done uploading segments... Dec 28 17:52:35.722769386 (554h23m35.593852431s) [upload=486048]: Upload finished. err: <nil> Duration: 5.695460547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X..X.............X.........................] duration: 3.480262816s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:53:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.480289317s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X........X..] duration: 39.479159841s trace: https://jaeger.dev.storj.io/trace/29b498816f392fc6 FAIL (2025-12-28 17:53:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6770948 (554h22m7.427853983s) [noise=true , address=77.90.31.13:4637 , piece_id=7IQITGZPXZVGGJE5WJUTIUPJQG7G6R2UXLQIEIDCSZPEFSZGO7GA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.556876499 (554h22m7.427959544s) [noise=true , address=145.79.227.44:4208 , piece_id=MX6FZVSTYGJCJ5DD3LXV36FRXMRTDY5VHPFZEKBNDYVNBEBSEPGQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557122898 (554h22m7.428205943s) [noise=true , address=77.90.31.16:4767 , piece_id=2IMQX3JLVFYYCQ6LKBLDVNDB6X3MMSTFG2YN6NZRCIR2QOS53HUQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557323139 (554h22m7.428406184s) [noise=true , address=145.79.227.25:4624 , piece_id=O57LSKM4YSSIA7N5K7VVP6XO6WQ4VQGZXVRQTBIRTVQP462SV65Q, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557456964 (554h22m7.428540009s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557722129 (554h22m7.428805165s) [noise=true , address=145.79.227.36:4549 , piece_id=TO6PZ3W33Q5EYTEEWOPYT52PFIRVJCNUTV2PNMWUGLHFBMTCJBGQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557973388 (554h22m7.429056423s) [noise=true , address=145.79.227.33:4882 , piece_id=TELMFJG4H5HSTI545736O452TWVRZ4OVVU54B3CEITUXWPRAOLHA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558373519 (554h22m7.429456564s) [noise=true , address=145.79.227.49:4342 , piece_id=X3LJ7ACO2LQTJBIGEQPIM2LY55ILNW6THFE6BWCZYGA2UKKKVFFQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558388057 (554h22m7.429471092s) [noise=true , address=145.79.227.29:4969 , piece_id=SOEYSHES5BYEGPBQAS24MCKRHH4ALL5FVEDUKKFIVWX3OAOD5Y6A, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558791585 (554h22m7.42987463s) [noise=true , address=145.79.227.31:4809 , piece_id=WYJ2EU3QN3FFZFL5QTP5CNAS5BCIUIFMH63C54ZLQJZY54TPRLVA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558823486 (554h22m7.429906531s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559230000 (554h22m7.430313035s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559247845 (554h22m7.43033089s) [noise=true , address=145.79.227.21:4481 , piece_id=MEAY63J5KAYJJ4TJP2IGY2PC6NJEOVRZ24QY6TQD4ATGBURA5QCQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559656883 (554h22m7.430739918s) [noise=true , address=193.149.164.143:4702 , piece_id=RERLF6JJJ7DWQVYJDSDET5OOWHADUSJXWL2VJJI56JINOZIGDHWA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.562207000 (554h22m7.433290055s) [noise=true , address=64.49.13.140:4516 , piece_id=72KQD7KPLSQUNDS3ILVTZTSXGQ6UF7MD3P2CUJ5CL7IAC2MFSI7A, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.564270158 (554h22m7.435353313s) [noise=true , address=45.91.116.5:4312 , piece_id=ZNPVKTONLL6NFMHUNS53ERZCRIJMA3XISFSS2NBEZOKKWQZO6ECQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:08.488386820 (554h22m8.359469855s) [noise=true , address=193.149.164.143:4702 , piece_id=RERLF6JJJ7DWQVYJDSDET5OOWHADUSJXWL2VJJI56JINOZIGDHWA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 28 17:51:08.868542955 (554h22m8.73962599s) [upload=486027]: Next returned nil segment Dec 28 17:51:08.868558364 (554h22m8.739641409s) [upload=486027]: Waiting for error group managing segments... Dec 28 17:51:09.298510128 (554h22m9.169593163s) [noise=true , address=64.49.13.137:4503 , piece_id=WZZ36TJ4V76JHORSPNBADMDSL6YYO3EW4DEQKDK3VIMZ4F3G7JSQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.396193335 (554h22m9.26727639s) [noise=true , address=64.49.13.140:4516 , piece_id=72KQD7KPLSQUNDS3ILVTZTSXGQ6UF7MD3P2CUJ5CL7IAC2MFSI7A, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.426697790 (554h22m9.297780835s) [noise=true , address=145.79.227.36:4549 , piece_id=TO6PZ3W33Q5EYTEEWOPYT52PFIRVJCNUTV2PNMWUGLHFBMTCJBGQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.441962316 (554h22m9.313045361s) [noise=true , address=145.79.227.31:4809 , piece_id=WYJ2EU3QN3FFZFL5QTP5CNAS5BCIUIFMH63C54ZLQJZY54TPRLVA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.443532056 (554h22m9.314615101s) [noise=true , address=145.79.227.25:4624 , piece_id=O57LSKM4YSSIA7N5K7VVP6XO6WQ4VQGZXVRQTBIRTVQP462SV65Q, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.448848529 (554h22m9.319931574s) [noise=true , address=145.79.227.21:4481 , piece_id=MEAY63J5KAYJJ4TJP2IGY2PC6NJEOVRZ24QY6TQD4ATGBURA5QCQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.449293015 (554h22m9.32037605s) [noise=true , address=145.79.227.29:4969 , piece_id=SOEYSHES5BYEGPBQAS24MCKRHH4ALL5FVEDUKKFIVWX3OAOD5Y6A, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.520315725 (554h22m9.391398771s) [noise=true , address=145.79.227.19:4100 , piece_id=X4NAGMU2TFRKKTTETO7AV2LJO2BPDI6TPH32NQFWGNYUAXN3UDRQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.645133701 (554h22m9.516216746s) [noise=true , address=77.90.31.13:4637 , piece_id=7IQITGZPXZVGGJE5WJUTIUPJQG7G6R2UXLQIEIDCSZPEFSZGO7GA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.670952274 (554h22m9.542035309s) [noise=true , address=45.91.116.5:4312 , piece_id=ZNPVKTONLL6NFMHUNS53ERZCRIJMA3XISFSS2NBEZOKKWQZO6ECQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.693198264 (554h22m9.564281309s) [noise=true , address=77.90.31.16:4767 , piece_id=2IMQX3JLVFYYCQ6LKBLDVNDB6X3MMSTFG2YN6NZRCIR2QOS53HUQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.298273265 (554h22m10.16935631s) [noise=true , address=145.79.227.33:4882 , piece_id=TELMFJG4H5HSTI545736O452TWVRZ4OVVU54B3CEITUXWPRAOLHA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.377531416 (554h22m10.248614451s) [noise=true , address=145.79.227.49:4342 , piece_id=X3LJ7ACO2LQTJBIGEQPIM2LY55ILNW6THFE6BWCZYGA2UKKKVFFQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.382353338 (554h22m10.253436373s) [noise=true , address=145.79.227.44:4208 , piece_id=MX6FZVSTYGJCJ5DD3LXV36FRXMRTDY5VHPFZEKBNDYVNBEBSEPGQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.382374157 (554h22m10.253457202s) [seg_pos={0 0}, upload=486027]: Segment reached optimal threshold of 14 pieces. Dec 28 17:51:10.382598706 (554h22m10.253681751s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection; write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection Dec 28 17:51:10.382606851 (554h22m10.253689896s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382812302 (554h22m10.253895337s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection; write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection Dec 28 17:51:10.382819997 (554h22m10.253903032s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382942811 (554h22m10.254025846s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 28 17:51:10.382948151 (554h22m10.254031186s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382963951 (554h22m10.254046996s) [seg_pos={0 0}, upload=486027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:51:10.383031409 (554h22m10.254114454s) [upload=486027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:51:10.383035807 (554h22m10.254118852s) [upload=486027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:51:10.688025582 (554h22m10.559108617s) [upload=486027]: Done uploading segments... Dec 28 17:51:10.688114761 (554h22m10.559197796s) [upload=486027]: Upload finished. err: <nil> Duration: 39.479189427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........X..XX.X.XX] duration: 30.003442957s trace: https://jaeger.dev.storj.io/trace/21f865b8a1414caa FAIL (2025-12-28 17:53:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ments... Dec 28 17:52:32.803342801 (554h23m32.674425836s) [upload=486048]: Got next segment. Inline: false Dec 28 17:52:32.803387345 (554h23m32.67447039s) [upload=486048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:52:32.803392775 (554h23m32.674475811s) [upload=486048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:52:33.045334645 (554h23m32.91641768s) [seg_pos={0 0}, upload=486048]: Begin upload segment... Dec 28 17:52:33.045392225 (554h23m32.91647526s) [seg_pos={0 0}, upload=486048]: Done begin upload segment. Dec 28 17:52:33.045418886 (554h23m32.916501921s) [noise=true , address=145.79.227.51:4410 , piece_id=CWQ3CLCDYYSAKX5LJ74ZPNBN6DNUOEGRK54ZWZREK2PFGQQ45J4A, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.045540017 (554h23m32.916623052s) [noise=true , address=145.79.227.16:4909 , piece_id=37PO6YVMUFLFW4W5WPFPLBMB6PMFHURXMR77A65QWZUEDIVTMJQQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.046097789 (554h23m32.917180824s) [noise=true , address=45.91.116.6:4976 , piece_id=7QMUCDH5PKQ7D4U6CXTZMZ4HWJPZH5WZ7AXX5FJKCCGM5IBD7SMA, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.046386147 (554h23m32.917469182s) [noise=true , address=145.79.227.13:4803 , piece_id=FQX4NPB2ROF2YQK7QN7PYYCUBDMRJRN6H37IHJXUU3O6JMTRY5NQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.046622327 (554h23m32.917705372s) [noise=true , address=145.79.227.50:4375 , piece_id=HBXUI7G4HUDSS6SOOJHIYQDDKPAPF3A2Y7JBXBMQ2YKGQ32V4CXQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.046677433 (554h23m32.917760468s) [noise=true , address=77.90.31.13:4629 , piece_id=TD462EOSCOASQSBLJMBEGWJQMMZ2EOEOJCOUISL75JVE452VIKLQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.046986792 (554h23m32.918069837s) [noise=true , address=77.90.31.19:4841 , piece_id=WJKDMHYCOYBNH77VSRL5TECRGBV35X3WKVVKS7ANBF5KIBX6V5NA, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.047322410 (554h23m32.918405455s) [noise=true , address=145.79.227.47:4267 , piece_id=GA3NJPN3TUS2C5VKYMHXPER77JTMBDFECBYXGU3JOST6KYULMBSQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.047347208 (554h23m32.918430253s) [noise=true , address=145.79.227.52:4447 , piece_id=IYIN2CIIO3LH3N32DDKDDNO4NCAYHKZV6JHAFOOBVVKIAXBEHKLQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.047663209 (554h23m32.918746245s) [noise=true , address=145.79.227.22:4510 , piece_id=UORFHYUBO3MOR4XJMQLDMW3CJIH4BGNAG5OTYRJLQU7NPMUZZ7EQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.047928465 (554h23m32.91901151s) [noise=true , address=45.91.116.14:4192 , piece_id=2WO7WIZTI5VUXNU5ZN6Y5OM7IQ4BQAB4A3LMUFBPF4DGIPCTYFUQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.048571349 (554h23m32.919654394s) [noise=true , address=145.79.227.2:4114 , piece_id=ZK4IOQNIWABK5U5UJJXUJUZFNFUHGCX2TXJTS2DIHQ7AXEBZ3FEQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.049213672 (554h23m32.920296717s) [noise=true , address=45.91.116.10:4081 , piece_id=5WSM23ARPD2LSJCL5DXDZASUYNXXOJZW5G2BH6JDRXXTYVBMVHCQ, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.049942821 (554h23m32.921025856s) [noise=true , address=145.79.227.45:4231 , piece_id=WDPMNFMIVW554T6KWK4A3Z2OAUDBU6UKLETYC3YGJKJGFDRNZDEA, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.050606705 (554h23m32.92168975s) [noise=true , address=145.79.227.26:4653 , piece_id=P7OBLIBIUDIM5CJYABXIDA2HWWCVR7UCHZF5CS5SCRX3OHBNCCKA, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:33.051595838 (554h23m32.922678883s) [noise=true , address=45.91.116.5:4298 , piece_id=PAR6V2V2GLCMEVXSIVNSRGJFV4VXYBTOA2MVAZWI5RUNSATHQOOA, seg_pos={0 0}, upload=486048]: Uploading piece... Dec 28 17:52:34.760797962 (554h23m34.631880997s) [upload=486048]: Next returned nil segment Dec 28 17:52:34.760813341 (554h23m34.631896386s) [upload=486048]: Waiting for error group managing segments... Dec 28 17:52:35.046758622 (554h23m34.917841657s) [noise=true , address=145.79.227.22:4510 , piece_id=UORFHYUBO3MOR4XJMQLDMW3CJIH4BGNAG5OTYRJLQU7NPMUZZ7EQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.050191930 (554h23m34.921274975s) [noise=true , address=145.79.227.51:4410 , piece_id=CWQ3CLCDYYSAKX5LJ74ZPNBN6DNUOEGRK54ZWZREK2PFGQQ45J4A, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.056530950 (554h23m34.927613985s) [noise=true , address=145.79.227.50:4375 , piece_id=HBXUI7G4HUDSS6SOOJHIYQDDKPAPF3A2Y7JBXBMQ2YKGQ32V4CXQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.065906543 (554h23m34.936989578s) [noise=true , address=145.79.227.2:4114 , piece_id=ZK4IOQNIWABK5U5UJJXUJUZFNFUHGCX2TXJTS2DIHQ7AXEBZ3FEQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.066317215 (554h23m34.93740027s) [noise=true , address=145.79.227.13:4803 , piece_id=FQX4NPB2ROF2YQK7QN7PYYCUBDMRJRN6H37IHJXUU3O6JMTRY5NQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.070497685 (554h23m34.94158072s) [noise=true , address=145.79.227.45:4231 , piece_id=WDPMNFMIVW554T6KWK4A3Z2OAUDBU6UKLETYC3YGJKJGFDRNZDEA, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.071050718 (554h23m34.942133753s) [noise=true , address=145.79.227.52:4447 , piece_id=IYIN2CIIO3LH3N32DDKDDNO4NCAYHKZV6JHAFOOBVVKIAXBEHKLQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.083315753 (554h23m34.954398788s) [noise=true , address=145.79.227.16:4909 , piece_id=37PO6YVMUFLFW4W5WPFPLBMB6PMFHURXMR77A65QWZUEDIVTMJQQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.100352975 (554h23m34.97143602s) [noise=true , address=145.79.227.47:4267 , piece_id=GA3NJPN3TUS2C5VKYMHXPER77JTMBDFECBYXGU3JOST6KYULMBSQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.306619943 (554h23m35.177702979s) [noise=true , address=77.90.31.19:4841 , piece_id=WJKDMHYCOYBNH77VSRL5TECRGBV35X3WKVVKS7ANBF5KIBX6V5NA, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.312275022 (554h23m35.183358057s) [noise=true , address=45.91.116.14:4192 , piece_id=2WO7WIZTI5VUXNU5ZN6Y5OM7IQ4BQAB4A3LMUFBPF4DGIPCTYFUQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.390502781 (554h23m35.261585826s) [noise=true , address=45.91.116.10:4081 , piece_id=5WSM23ARPD2LSJCL5DXDZASUYNXXOJZW5G2BH6JDRXXTYVBMVHCQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.431665262 (554h23m35.302748307s) [noise=true , address=45.91.116.5:4298 , piece_id=PAR6V2V2GLCMEVXSIVNSRGJFV4VXYBTOA2MVAZWI5RUNSATHQOOA, seg_pos={0 0}, upload=486048]: Done uploading piece. err: <nil> Dec 28 17:52:35.431691231 (554h23m35.302774276s) [seg_pos={0 0}, upload=486048]: Segment reached optimal threshold of 13 pieces. Dec 28 17:52:35.431926018 (554h23m35.303009063s) [noise=true , address=145.79.227.26:4653 , piece_id=P7OBLIBIUDIM5CJYABXIDA2HWWCVR7UCHZF5CS5SCRX3OHBNCCKA, seg_pos={0 0}, upload=486048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:45356->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:45356->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45356->145.79.227.26:4653: use of closed network connection Dec 28 17:52:35.431935205 (554h23m35.303018241s) [noise=true , address=145.79.227.26:4653 , piece_id=P7OBLIBIUDIM5CJYABXIDA2HWWCVR7UCHZF5CS5SCRX3OHBNCCKA, seg_pos={0 0}, upload=486048]: Long tail cancellation due to optimal threshold Dec 28 17:52:35.432249043 (554h23m35.303332088s) [noise=true , address=77.90.31.13:4629 , piece_id=TD462EOSCOASQSBLJMBEGWJQMMZ2EOEOJCOUISL75JVE452VIKLQ, seg_pos={0 0}, upload=486048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:34960->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:34960->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34960->77.90.31.13:4629: use of closed network connection Dec 28 17:52:35.432258261 (554h23m35.303341296s) [noise=true , address=77.90.31.13:4629 , piece_id=TD462EOSCOASQSBLJMBEGWJQMMZ2EOEOJCOUISL75JVE452VIKLQ, seg_pos={0 0}, upload=486048]: Long tail cancellation due to optimal threshold Dec 28 17:52:35.432495253 (554h23m35.303578298s) [noise=true , address=45.91.116.6:4976 , piece_id=7QMUCDH5PKQ7D4U6CXTZMZ4HWJPZH5WZ7AXX5FJKCCGM5IBD7SMA, seg_pos={0 0}, upload=486048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 17:52:35.432548824 (554h23m35.303631889s) [noise=true , address=45.91.116.6:4976 , piece_id=7QMUCDH5PKQ7D4U6CXTZMZ4HWJPZH5WZ7AXX5FJKCCGM5IBD7SMA, seg_pos={0 0}, upload=486048]: Long tail cancellation due to optimal threshold Dec 28 17:52:35.432574994 (554h23m35.303658039s) [seg_pos={0 0}, upload=486048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:52:35.432658924 (554h23m35.303741969s) [upload=486048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:52:35.432665676 (554h23m35.303748722s) [upload=486048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:52:35.722617406 (554h23m35.593700451s) [upload=486048]: Done uploading segments... Dec 28 17:52:35.722769386 (554h23m35.593852431s) [upload=486048]: Upload finished. err: <nil> Duration: 30.003456822s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X..X.............X........................] duration: 3.420924757s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:53:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42095276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X........X.] duration: 15.476727614s trace: https://jaeger.dev.storj.io/trace/29b498816f392fc6 FAIL (2025-12-28 17:52:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6770948 (554h22m7.427853983s) [noise=true , address=77.90.31.13:4637 , piece_id=7IQITGZPXZVGGJE5WJUTIUPJQG7G6R2UXLQIEIDCSZPEFSZGO7GA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.556876499 (554h22m7.427959544s) [noise=true , address=145.79.227.44:4208 , piece_id=MX6FZVSTYGJCJ5DD3LXV36FRXMRTDY5VHPFZEKBNDYVNBEBSEPGQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557122898 (554h22m7.428205943s) [noise=true , address=77.90.31.16:4767 , piece_id=2IMQX3JLVFYYCQ6LKBLDVNDB6X3MMSTFG2YN6NZRCIR2QOS53HUQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557323139 (554h22m7.428406184s) [noise=true , address=145.79.227.25:4624 , piece_id=O57LSKM4YSSIA7N5K7VVP6XO6WQ4VQGZXVRQTBIRTVQP462SV65Q, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557456964 (554h22m7.428540009s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557722129 (554h22m7.428805165s) [noise=true , address=145.79.227.36:4549 , piece_id=TO6PZ3W33Q5EYTEEWOPYT52PFIRVJCNUTV2PNMWUGLHFBMTCJBGQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557973388 (554h22m7.429056423s) [noise=true , address=145.79.227.33:4882 , piece_id=TELMFJG4H5HSTI545736O452TWVRZ4OVVU54B3CEITUXWPRAOLHA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558373519 (554h22m7.429456564s) [noise=true , address=145.79.227.49:4342 , piece_id=X3LJ7ACO2LQTJBIGEQPIM2LY55ILNW6THFE6BWCZYGA2UKKKVFFQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558388057 (554h22m7.429471092s) [noise=true , address=145.79.227.29:4969 , piece_id=SOEYSHES5BYEGPBQAS24MCKRHH4ALL5FVEDUKKFIVWX3OAOD5Y6A, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558791585 (554h22m7.42987463s) [noise=true , address=145.79.227.31:4809 , piece_id=WYJ2EU3QN3FFZFL5QTP5CNAS5BCIUIFMH63C54ZLQJZY54TPRLVA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558823486 (554h22m7.429906531s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559230000 (554h22m7.430313035s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559247845 (554h22m7.43033089s) [noise=true , address=145.79.227.21:4481 , piece_id=MEAY63J5KAYJJ4TJP2IGY2PC6NJEOVRZ24QY6TQD4ATGBURA5QCQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559656883 (554h22m7.430739918s) [noise=true , address=193.149.164.143:4702 , piece_id=RERLF6JJJ7DWQVYJDSDET5OOWHADUSJXWL2VJJI56JINOZIGDHWA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.562207000 (554h22m7.433290055s) [noise=true , address=64.49.13.140:4516 , piece_id=72KQD7KPLSQUNDS3ILVTZTSXGQ6UF7MD3P2CUJ5CL7IAC2MFSI7A, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.564270158 (554h22m7.435353313s) [noise=true , address=45.91.116.5:4312 , piece_id=ZNPVKTONLL6NFMHUNS53ERZCRIJMA3XISFSS2NBEZOKKWQZO6ECQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:08.488386820 (554h22m8.359469855s) [noise=true , address=193.149.164.143:4702 , piece_id=RERLF6JJJ7DWQVYJDSDET5OOWHADUSJXWL2VJJI56JINOZIGDHWA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 28 17:51:08.868542955 (554h22m8.73962599s) [upload=486027]: Next returned nil segment Dec 28 17:51:08.868558364 (554h22m8.739641409s) [upload=486027]: Waiting for error group managing segments... Dec 28 17:51:09.298510128 (554h22m9.169593163s) [noise=true , address=64.49.13.137:4503 , piece_id=WZZ36TJ4V76JHORSPNBADMDSL6YYO3EW4DEQKDK3VIMZ4F3G7JSQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.396193335 (554h22m9.26727639s) [noise=true , address=64.49.13.140:4516 , piece_id=72KQD7KPLSQUNDS3ILVTZTSXGQ6UF7MD3P2CUJ5CL7IAC2MFSI7A, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.426697790 (554h22m9.297780835s) [noise=true , address=145.79.227.36:4549 , piece_id=TO6PZ3W33Q5EYTEEWOPYT52PFIRVJCNUTV2PNMWUGLHFBMTCJBGQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.441962316 (554h22m9.313045361s) [noise=true , address=145.79.227.31:4809 , piece_id=WYJ2EU3QN3FFZFL5QTP5CNAS5BCIUIFMH63C54ZLQJZY54TPRLVA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.443532056 (554h22m9.314615101s) [noise=true , address=145.79.227.25:4624 , piece_id=O57LSKM4YSSIA7N5K7VVP6XO6WQ4VQGZXVRQTBIRTVQP462SV65Q, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.448848529 (554h22m9.319931574s) [noise=true , address=145.79.227.21:4481 , piece_id=MEAY63J5KAYJJ4TJP2IGY2PC6NJEOVRZ24QY6TQD4ATGBURA5QCQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.449293015 (554h22m9.32037605s) [noise=true , address=145.79.227.29:4969 , piece_id=SOEYSHES5BYEGPBQAS24MCKRHH4ALL5FVEDUKKFIVWX3OAOD5Y6A, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.520315725 (554h22m9.391398771s) [noise=true , address=145.79.227.19:4100 , piece_id=X4NAGMU2TFRKKTTETO7AV2LJO2BPDI6TPH32NQFWGNYUAXN3UDRQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.645133701 (554h22m9.516216746s) [noise=true , address=77.90.31.13:4637 , piece_id=7IQITGZPXZVGGJE5WJUTIUPJQG7G6R2UXLQIEIDCSZPEFSZGO7GA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.670952274 (554h22m9.542035309s) [noise=true , address=45.91.116.5:4312 , piece_id=ZNPVKTONLL6NFMHUNS53ERZCRIJMA3XISFSS2NBEZOKKWQZO6ECQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.693198264 (554h22m9.564281309s) [noise=true , address=77.90.31.16:4767 , piece_id=2IMQX3JLVFYYCQ6LKBLDVNDB6X3MMSTFG2YN6NZRCIR2QOS53HUQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.298273265 (554h22m10.16935631s) [noise=true , address=145.79.227.33:4882 , piece_id=TELMFJG4H5HSTI545736O452TWVRZ4OVVU54B3CEITUXWPRAOLHA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.377531416 (554h22m10.248614451s) [noise=true , address=145.79.227.49:4342 , piece_id=X3LJ7ACO2LQTJBIGEQPIM2LY55ILNW6THFE6BWCZYGA2UKKKVFFQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.382353338 (554h22m10.253436373s) [noise=true , address=145.79.227.44:4208 , piece_id=MX6FZVSTYGJCJ5DD3LXV36FRXMRTDY5VHPFZEKBNDYVNBEBSEPGQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.382374157 (554h22m10.253457202s) [seg_pos={0 0}, upload=486027]: Segment reached optimal threshold of 14 pieces. Dec 28 17:51:10.382598706 (554h22m10.253681751s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection; write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection Dec 28 17:51:10.382606851 (554h22m10.253689896s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382812302 (554h22m10.253895337s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection; write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection Dec 28 17:51:10.382819997 (554h22m10.253903032s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382942811 (554h22m10.254025846s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 28 17:51:10.382948151 (554h22m10.254031186s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382963951 (554h22m10.254046996s) [seg_pos={0 0}, upload=486027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:51:10.383031409 (554h22m10.254114454s) [upload=486027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:51:10.383035807 (554h22m10.254118852s) [upload=486027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:51:10.688025582 (554h22m10.559108617s) [upload=486027]: Done uploading segments... Dec 28 17:51:10.688114761 (554h22m10.559197796s) [upload=486027]: Upload finished. err: <nil> Duration: 15.47677836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [....XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........X..XX.X.X] duration: 30.004024813s trace: https://jaeger.dev.storj.io/trace/2a0295cde0e96002 FAIL (2025-12-28 17:52:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 3BXL4FLDS3WVP6M6CJGLUN46QPD4MIFH5PO3EYFB3HF4RADIRAEQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: ERCODUPEKRLMKXHWB2ASBOGQ24PFDNSVXAYH62CJXGQS6ZZK2IRA) context deadline exceeded ======LOGS====== Dec 28 17:52:02.596102789 (554h23m2.467185834s) [upload=486038]: Starting upload Dec 28 17:52:02.596183643 (554h23m2.467266678s) [upload=486038]: Done starting upload Dec 28 17:52:02.596396448 (554h23m2.467479493s) [upload=486038]: Uploading segments... Dec 28 17:52:02.596507079 (554h23m2.467590124s) [upload=486038]: Got next segment. Inline: false Dec 28 17:52:02.596546664 (554h23m2.467629709s) [upload=486038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:52:02.596551463 (554h23m2.467634498s) [upload=486038]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:52:02.848222459 (554h23m2.719305514s) [seg_pos={0 0}, upload=486038]: Begin upload segment... Dec 28 17:52:02.848287273 (554h23m2.719370318s) [seg_pos={0 0}, upload=486038]: Done begin upload segment. Dec 28 17:52:02.848320897 (554h23m2.719403932s) [noise=true , address=145.79.227.47:4265 , piece_id=N32IGA6QYGMR2XRMDZDGUIXUCCVP2Z6YUB2TAJZGC4IR5NUY43MA, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.848352367 (554h23m2.719435402s) [noise=true , address=77.90.31.19:4844 , piece_id=CYYP4ABG4MW3F57XFOQR55XNCZFDB4Q6Q2NMLF56SK6Q6BVGDQBA, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.848907494 (554h23m2.719990539s) [noise=true , address=45.91.116.7:4009 , piece_id=LYNRUSAKOZOGBV7KHQK7ZIL2G6OSEI54VMESQAXKMGUB67CDCT2Q, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.849369604 (554h23m2.720452649s) [noise=true , address=145.79.227.19:4078 , piece_id=UFWE5JEY2IUOOPM3TIWQ7M6PZXMBEXWNAM65L4NG2DTZRR2GBN3A, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.849937194 (554h23m2.721020239s) [noise=true , address=77.90.31.18:4805 , piece_id=TRP2KSRJEFKDABYCNGYHIRHMPFDJKIJ4L3APISSITVGP4WNZKKMQ, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.851300270 (554h23m2.722383315s) [noise=true , address=77.90.31.11:4556 , piece_id=J7XQESQWDI4LNTE5HCUE46NLNLML3D72G6G7EQUD3RJNQSJ7U5GQ, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.851592517 (554h23m2.722675552s) [noise=true , address=145.79.227.32:4836 , piece_id=Q3BCNPD7QQVNM7E67FXQDN56C6FBSDK6EK2JZOZ6GV4HYUG26FLQ, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.852066569 (554h23m2.723149604s) [noise=true , address=145.79.227.15:4874 , piece_id=4MZRREAQUIBCKPCXF5MHPIHQZWJ32UGYAGMANNOHE24BHC7XYDEQ, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.852508851 (554h23m2.723591896s) [noise=true , address=45.91.116.10:4083 , piece_id=N7KAWJPMOLD3G72C4SJFMZQEX5473LUE2J2ADTG42LHEXSROGTIA, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.852957616 (554h23m2.724040661s) [noise=true , address=145.79.227.35:4481 , piece_id=IQKW4P2NWLGSOR4SGZH5K4NPEZBHFRPTXXRUDSW4IBCB5OTKIGLA, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.853391562 (554h23m2.724474607s) [noise=true , address=45.91.116.14:4190 , piece_id=H5KA2C73ZFTHWFFUD4TQJ6PWD5A2QMZYS66VSRDZ6APMQNYYNTJQ, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.853778539 (554h23m2.724861585s) [noise=true , address=145.79.227.11:4730 , piece_id=HIGA6LFYGZHXPB53EAOKFTLJPSSWFWBKK4QCNIFEOGNUWMOKRXIQ, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.854330600 (554h23m2.725413635s) [noise=true , address=45.91.116.8:4046 , piece_id=3BXL4FLDS3WVP6M6CJGLUN46QPD4MIFH5PO3EYFB3HF4RADIRAEQ, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.854778062 (554h23m2.725861097s) [noise=true , address=145.79.227.36:4518 , piece_id=Y2TDUIDQPQO5D7DSXJDNCTJ6JWZKI4ASM3BXHHIS3YCEKWP4QJIA, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.856076956 (554h23m2.727160001s) [noise=true , address=145.79.227.8:4329 , piece_id=32ZNXEVHYWF2LLOPKBIVXBFT2DFW4B2PBQHDOTGCWAFPS37WLMZQ, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:02.857481971 (554h23m2.728565006s) [noise=true , address=45.91.116.9:4264 , piece_id=ERCODUPEKRLMKXHWB2ASBOGQ24PFDNSVXAYH62CJXGQS6ZZK2IRA, seg_pos={0 0}, upload=486038]: Uploading piece... Dec 28 17:52:04.564235792 (554h23m4.435318838s) [upload=486038]: Next returned nil segment Dec 28 17:52:04.564253406 (554h23m4.435336452s) [upload=486038]: Waiting for error group managing segments... Dec 28 17:52:04.849341124 (554h23m4.720424169s) [noise=true , address=145.79.227.19:4078 , piece_id=UFWE5JEY2IUOOPM3TIWQ7M6PZXMBEXWNAM65L4NG2DTZRR2GBN3A, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:04.861727159 (554h23m4.732810204s) [noise=true , address=145.79.227.15:4874 , piece_id=4MZRREAQUIBCKPCXF5MHPIHQZWJ32UGYAGMANNOHE24BHC7XYDEQ, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:04.868947778 (554h23m4.740030813s) [noise=true , address=145.79.227.36:4518 , piece_id=Y2TDUIDQPQO5D7DSXJDNCTJ6JWZKI4ASM3BXHHIS3YCEKWP4QJIA, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:04.885702363 (554h23m4.756785418s) [noise=true , address=145.79.227.47:4265 , piece_id=N32IGA6QYGMR2XRMDZDGUIXUCCVP2Z6YUB2TAJZGC4IR5NUY43MA, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:04.892819444 (554h23m4.763902489s) [noise=true , address=145.79.227.11:4730 , piece_id=HIGA6LFYGZHXPB53EAOKFTLJPSSWFWBKK4QCNIFEOGNUWMOKRXIQ, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:04.895398575 (554h23m4.76648162s) [noise=true , address=145.79.227.8:4329 , piece_id=32ZNXEVHYWF2LLOPKBIVXBFT2DFW4B2PBQHDOTGCWAFPS37WLMZQ, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:04.904815066 (554h23m4.775898112s) [noise=true , address=145.79.227.32:4836 , piece_id=Q3BCNPD7QQVNM7E67FXQDN56C6FBSDK6EK2JZOZ6GV4HYUG26FLQ, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:05.044293127 (554h23m4.915376172s) [noise=true , address=77.90.31.11:4556 , piece_id=J7XQESQWDI4LNTE5HCUE46NLNLML3D72G6G7EQUD3RJNQSJ7U5GQ, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:05.047104881 (554h23m4.918187916s) [noise=true , address=77.90.31.18:4805 , piece_id=TRP2KSRJEFKDABYCNGYHIRHMPFDJKIJ4L3APISSITVGP4WNZKKMQ, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:05.146784509 (554h23m5.017867544s) [noise=true , address=77.90.31.19:4844 , piece_id=CYYP4ABG4MW3F57XFOQR55XNCZFDB4Q6Q2NMLF56SK6Q6BVGDQBA, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:05.800533794 (554h23m5.671616839s) [noise=true , address=45.91.116.10:4083 , piece_id=N7KAWJPMOLD3G72C4SJFMZQEX5473LUE2J2ADTG42LHEXSROGTIA, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:05.851648235 (554h23m5.72273127s) [noise=true , address=45.91.116.8:4046 , piece_id=3BXL4FLDS3WVP6M6CJGLUN46QPD4MIFH5PO3EYFB3HF4RADIRAEQ, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:05.882982591 (554h23m5.754065626s) [noise=true , address=45.91.116.9:4264 , piece_id=ERCODUPEKRLMKXHWB2ASBOGQ24PFDNSVXAYH62CJXGQS6ZZK2IRA, seg_pos={0 0}, upload=486038]: Done uploading piece. err: <nil> Dec 28 17:52:05.883007989 (554h23m5.754091034s) [seg_pos={0 0}, upload=486038]: Segment reached optimal threshold of 13 pieces. Dec 28 17:52:05.883232156 (554h23m5.754315201s) [noise=true , address=45.91.116.7:4009 , piece_id=LYNRUSAKOZOGBV7KHQK7ZIL2G6OSEI54VMESQAXKMGUB67CDCT2Q, seg_pos={0 0}, upload=486038]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 17:52:05.883241304 (554h23m5.754324359s) [noise=true , address=45.91.116.7:4009 , piece_id=LYNRUSAKOZOGBV7KHQK7ZIL2G6OSEI54VMESQAXKMGUB67CDCT2Q, seg_pos={0 0}, upload=486038]: Long tail cancellation due to optimal threshold Dec 28 17:52:05.883394475 (554h23m5.75447751s) [noise=true , address=145.79.227.35:4481 , piece_id=IQKW4P2NWLGSOR4SGZH5K4NPEZBHFRPTXXRUDSW4IBCB5OTKIGLA, seg_pos={0 0}, upload=486038]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 17:52:05.883404895 (554h23m5.75448794s) [noise=true , address=145.79.227.35:4481 , piece_id=IQKW4P2NWLGSOR4SGZH5K4NPEZBHFRPTXXRUDSW4IBCB5OTKIGLA, seg_pos={0 0}, upload=486038]: Long tail cancellation due to optimal threshold Dec 28 17:52:05.883429141 (554h23m5.754512176s) [noise=true , address=45.91.116.14:4190 , piece_id=H5KA2C73ZFTHWFFUD4TQJ6PWD5A2QMZYS66VSRDZ6APMQNYYNTJQ, seg_pos={0 0}, upload=486038]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 17:52:05.883434100 (554h23m5.754517136s) [noise=true , address=45.91.116.14:4190 , piece_id=H5KA2C73ZFTHWFFUD4TQJ6PWD5A2QMZYS66VSRDZ6APMQNYYNTJQ, seg_pos={0 0}, upload=486038]: Long tail cancellation due to optimal threshold Dec 28 17:52:05.883448288 (554h23m5.754531333s) [seg_pos={0 0}, upload=486038]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:52:05.883566062 (554h23m5.754649097s) [upload=486038]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:52:05.883571803 (554h23m5.754654838s) [upload=486038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:52:06.178037198 (554h23m6.049120233s) [upload=486038]: Done uploading segments... Dec 28 17:52:06.178057916 (554h23m6.049140952s) [upload=486038]: Upload finished. err: <nil> Duration: 30.004053167s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X..X.............X.......................] duration: 3.540471345s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:52:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.540501743s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X........X] duration: 1m0.004317783s trace: https://jaeger.dev.storj.io/trace/29b498816f392fc6 FAIL (2025-12-28 17:52:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6770948 (554h22m7.427853983s) [noise=true , address=77.90.31.13:4637 , piece_id=7IQITGZPXZVGGJE5WJUTIUPJQG7G6R2UXLQIEIDCSZPEFSZGO7GA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.556876499 (554h22m7.427959544s) [noise=true , address=145.79.227.44:4208 , piece_id=MX6FZVSTYGJCJ5DD3LXV36FRXMRTDY5VHPFZEKBNDYVNBEBSEPGQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557122898 (554h22m7.428205943s) [noise=true , address=77.90.31.16:4767 , piece_id=2IMQX3JLVFYYCQ6LKBLDVNDB6X3MMSTFG2YN6NZRCIR2QOS53HUQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557323139 (554h22m7.428406184s) [noise=true , address=145.79.227.25:4624 , piece_id=O57LSKM4YSSIA7N5K7VVP6XO6WQ4VQGZXVRQTBIRTVQP462SV65Q, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557456964 (554h22m7.428540009s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557722129 (554h22m7.428805165s) [noise=true , address=145.79.227.36:4549 , piece_id=TO6PZ3W33Q5EYTEEWOPYT52PFIRVJCNUTV2PNMWUGLHFBMTCJBGQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.557973388 (554h22m7.429056423s) [noise=true , address=145.79.227.33:4882 , piece_id=TELMFJG4H5HSTI545736O452TWVRZ4OVVU54B3CEITUXWPRAOLHA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558373519 (554h22m7.429456564s) [noise=true , address=145.79.227.49:4342 , piece_id=X3LJ7ACO2LQTJBIGEQPIM2LY55ILNW6THFE6BWCZYGA2UKKKVFFQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558388057 (554h22m7.429471092s) [noise=true , address=145.79.227.29:4969 , piece_id=SOEYSHES5BYEGPBQAS24MCKRHH4ALL5FVEDUKKFIVWX3OAOD5Y6A, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558791585 (554h22m7.42987463s) [noise=true , address=145.79.227.31:4809 , piece_id=WYJ2EU3QN3FFZFL5QTP5CNAS5BCIUIFMH63C54ZLQJZY54TPRLVA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.558823486 (554h22m7.429906531s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559230000 (554h22m7.430313035s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559247845 (554h22m7.43033089s) [noise=true , address=145.79.227.21:4481 , piece_id=MEAY63J5KAYJJ4TJP2IGY2PC6NJEOVRZ24QY6TQD4ATGBURA5QCQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.559656883 (554h22m7.430739918s) [noise=true , address=193.149.164.143:4702 , piece_id=RERLF6JJJ7DWQVYJDSDET5OOWHADUSJXWL2VJJI56JINOZIGDHWA, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.562207000 (554h22m7.433290055s) [noise=true , address=64.49.13.140:4516 , piece_id=72KQD7KPLSQUNDS3ILVTZTSXGQ6UF7MD3P2CUJ5CL7IAC2MFSI7A, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:07.564270158 (554h22m7.435353313s) [noise=true , address=45.91.116.5:4312 , piece_id=ZNPVKTONLL6NFMHUNS53ERZCRIJMA3XISFSS2NBEZOKKWQZO6ECQ, seg_pos={0 0}, upload=486027]: Uploading piece... Dec 28 17:51:08.488386820 (554h22m8.359469855s) [noise=true , address=193.149.164.143:4702 , piece_id=RERLF6JJJ7DWQVYJDSDET5OOWHADUSJXWL2VJJI56JINOZIGDHWA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 28 17:51:08.868542955 (554h22m8.73962599s) [upload=486027]: Next returned nil segment Dec 28 17:51:08.868558364 (554h22m8.739641409s) [upload=486027]: Waiting for error group managing segments... Dec 28 17:51:09.298510128 (554h22m9.169593163s) [noise=true , address=64.49.13.137:4503 , piece_id=WZZ36TJ4V76JHORSPNBADMDSL6YYO3EW4DEQKDK3VIMZ4F3G7JSQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.396193335 (554h22m9.26727639s) [noise=true , address=64.49.13.140:4516 , piece_id=72KQD7KPLSQUNDS3ILVTZTSXGQ6UF7MD3P2CUJ5CL7IAC2MFSI7A, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.426697790 (554h22m9.297780835s) [noise=true , address=145.79.227.36:4549 , piece_id=TO6PZ3W33Q5EYTEEWOPYT52PFIRVJCNUTV2PNMWUGLHFBMTCJBGQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.441962316 (554h22m9.313045361s) [noise=true , address=145.79.227.31:4809 , piece_id=WYJ2EU3QN3FFZFL5QTP5CNAS5BCIUIFMH63C54ZLQJZY54TPRLVA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.443532056 (554h22m9.314615101s) [noise=true , address=145.79.227.25:4624 , piece_id=O57LSKM4YSSIA7N5K7VVP6XO6WQ4VQGZXVRQTBIRTVQP462SV65Q, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.448848529 (554h22m9.319931574s) [noise=true , address=145.79.227.21:4481 , piece_id=MEAY63J5KAYJJ4TJP2IGY2PC6NJEOVRZ24QY6TQD4ATGBURA5QCQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.449293015 (554h22m9.32037605s) [noise=true , address=145.79.227.29:4969 , piece_id=SOEYSHES5BYEGPBQAS24MCKRHH4ALL5FVEDUKKFIVWX3OAOD5Y6A, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.520315725 (554h22m9.391398771s) [noise=true , address=145.79.227.19:4100 , piece_id=X4NAGMU2TFRKKTTETO7AV2LJO2BPDI6TPH32NQFWGNYUAXN3UDRQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.645133701 (554h22m9.516216746s) [noise=true , address=77.90.31.13:4637 , piece_id=7IQITGZPXZVGGJE5WJUTIUPJQG7G6R2UXLQIEIDCSZPEFSZGO7GA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.670952274 (554h22m9.542035309s) [noise=true , address=45.91.116.5:4312 , piece_id=ZNPVKTONLL6NFMHUNS53ERZCRIJMA3XISFSS2NBEZOKKWQZO6ECQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:09.693198264 (554h22m9.564281309s) [noise=true , address=77.90.31.16:4767 , piece_id=2IMQX3JLVFYYCQ6LKBLDVNDB6X3MMSTFG2YN6NZRCIR2QOS53HUQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.298273265 (554h22m10.16935631s) [noise=true , address=145.79.227.33:4882 , piece_id=TELMFJG4H5HSTI545736O452TWVRZ4OVVU54B3CEITUXWPRAOLHA, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.377531416 (554h22m10.248614451s) [noise=true , address=145.79.227.49:4342 , piece_id=X3LJ7ACO2LQTJBIGEQPIM2LY55ILNW6THFE6BWCZYGA2UKKKVFFQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.382353338 (554h22m10.253436373s) [noise=true , address=145.79.227.44:4208 , piece_id=MX6FZVSTYGJCJ5DD3LXV36FRXMRTDY5VHPFZEKBNDYVNBEBSEPGQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: <nil> Dec 28 17:51:10.382374157 (554h22m10.253457202s) [seg_pos={0 0}, upload=486027]: Segment reached optimal threshold of 14 pieces. Dec 28 17:51:10.382598706 (554h22m10.253681751s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection; write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44960->45.91.116.13:4167: use of closed network connection Dec 28 17:51:10.382606851 (554h22m10.253689896s) [noise=true , address=45.91.116.13:4167 , piece_id=NRRFD3M3Q3WSUKIWZPKWAPUUO6IFIWGCWRATOSUV5NYSNHDRM7UQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382812302 (554h22m10.253895337s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE): protocol: write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection; write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60606->64.49.13.26:4953: use of closed network connection Dec 28 17:51:10.382819997 (554h22m10.253903032s) [noise=true , address=64.49.13.26:4953 , piece_id=72SKGSDXR3XGVSKSZZN75X4ZOZ3TH4Z46MME74G2G4QJIUFEDAJQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382942811 (554h22m10.254025846s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 28 17:51:10.382948151 (554h22m10.254031186s) [noise=true , address=145.79.227.34:4919 , piece_id=UZOE3BVOHXIQDQYCLK5YNCV7DPGX5TM2O7MTB545O7IMTUJ7KRCQ, seg_pos={0 0}, upload=486027]: Long tail cancellation due to optimal threshold Dec 28 17:51:10.382963951 (554h22m10.254046996s) [seg_pos={0 0}, upload=486027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:51:10.383031409 (554h22m10.254114454s) [upload=486027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:51:10.383035807 (554h22m10.254118852s) [upload=486027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:51:10.688025582 (554h22m10.559108617s) [upload=486027]: Done uploading segments... Dec 28 17:51:10.688114761 (554h22m10.559197796s) [upload=486027]: Upload finished. err: <nil> Duration: 1m0.004344675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........X..XX.X.] duration: 6.197912314s trace: https://jaeger.dev.storj.io/trace/45b2db7853d34e40 FAIL (2025-12-28 17:51:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: CTU2J7R4ESM5XSU6VT5NHCICKLEC6SU4SAHWV3MUFPYOQYZPRILA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: IDEHD3OONBEK4CLFANB4S74TIXBTMOX2M7PWF3EPAQZD5IAKHYAA) context deadline exceeded ======LOGS====== Dec 28 17:50:22.937429599 (554h21m22.808512634s) [upload=486017]: Starting upload Dec 28 17:50:22.937504411 (554h21m22.808587456s) [upload=486017]: Done starting upload Dec 28 17:50:22.937676038 (554h21m22.808759083s) [upload=486017]: Uploading segments... Dec 28 17:50:22.937734419 (554h21m22.808817464s) [upload=486017]: Got next segment. Inline: false Dec 28 17:50:22.937766320 (554h21m22.808849355s) [upload=486017]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:50:22.937771109 (554h21m22.808854144s) [upload=486017]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:50:23.183050895 (554h21m23.05413396s) [seg_pos={0 0}, upload=486017]: Begin upload segment... Dec 28 17:50:23.183103265 (554h21m23.0541863s) [seg_pos={0 0}, upload=486017]: Done begin upload segment. Dec 28 17:50:23.183142199 (554h21m23.054225234s) [noise=true , address=145.79.227.7:4295 , piece_id=IL2ZNGGEO7SQHYNL3O26XAZH46GY5QZVY67VQ2TQNFFHF5UOZQBA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.183233053 (554h21m23.054316088s) [noise=true , address=145.79.227.15:4874 , piece_id=NH36TESII4A6SK3SUZSD4BNMUUE6WQZDZBCJYWBRDGFG55BTTAFA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.183845408 (554h21m23.054928453s) [noise=true , address=145.79.227.34:4907 , piece_id=ZIBHRDM442UFQN6XC7BKLATZ4LTQOHYDZF53BZH7PTRYZVFCGD6Q, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.184310444 (554h21m23.055393479s) [noise=true , address=145.79.227.9:4366 , piece_id=IGG66AYDEXFFL6CTXCX54WTM2D4VTQYK4L7CXUIIHZU6ZVXARDIA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.184793603 (554h21m23.055876648s) [noise=true , address=77.90.31.10:4518 , piece_id=JCYIENVACJ7WD7JPAOF7V7OTMZZ73MUOZ77RLTZVWHGD7PF6YG6A, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.185256886 (554h21m23.056339931s) [noise=true , address=145.79.227.38:4979 , piece_id=FO67DAJVGNS4YXTHGRUCALI7T32PHBFZYHRFL35TAN53VT2HDTTA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.185673670 (554h21m23.056756715s) [noise=true , address=145.79.227.36:4518 , piece_id=EJMLGB6YIGOOTKFK73GYMMPVPZDCWSVWP7IIIJ4T6MAWJFL3XZVA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.186225391 (554h21m23.057308426s) [noise=true , address=77.90.31.15:4697 , piece_id=VTFX4PUEB75FODWO2BEJIZMU2YCWEEXSGNYL5AL5KDBNKDWJD4GA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.187044229 (554h21m23.058127264s) [noise=true , address=145.79.227.47:4265 , piece_id=DCTC4E7PKYB7MFVQZF3CWIY2J7CW7VJSYKD57TA6KKOVV7MCR63A, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.188220469 (554h21m23.059303514s) [noise=true , address=45.91.116.6:4974 , piece_id=CTU2J7R4ESM5XSU6VT5NHCICKLEC6SU4SAHWV3MUFPYOQYZPRILA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.190587236 (554h21m23.061670281s) [noise=true , address=145.79.227.5:4223 , piece_id=W7PHJEJAEBTPV4HRDL63FUSHND5TCLJKHOGDOR3VWWWC2JDURVEQ, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.192530416 (554h21m23.063613451s) [noise=true , address=145.79.227.26:4654 , piece_id=NCSFRKRKWGBJVDBMIUJXERBGLHANUBBCSFEXGAAUNWTGKCKY7PMA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.193893822 (554h21m23.064976867s) [noise=true , address=145.79.227.48:4303 , piece_id=HES7TTXHVQHVM5OGZH23HZGSAFJDQGY6NV7RBPRCKKWE535YLLGQ, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.195454204 (554h21m23.066537249s) [noise=true , address=77.90.31.19:4841 , piece_id=IDEHD3OONBEK4CLFANB4S74TIXBTMOX2M7PWF3EPAQZD5IAKHYAA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.196706198 (554h21m23.067789233s) [noise=true , address=145.79.227.4:4185 , piece_id=U5TWRN4VEWNCAJSKBCENXBOW3VB6MEYXPQLG65OCR6G2HDOQRI3Q, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.197426038 (554h21m23.068509083s) [noise=true , address=145.79.227.41:4086 , piece_id=4TMCGC5A5HDOVGI6HPYKEWXBNYEW76A4M24XRAEIFIQWPRSK4VLA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:24.855859198 (554h21m24.726942244s) [upload=486017]: Next returned nil segment Dec 28 17:50:24.855879107 (554h21m24.726962152s) [upload=486017]: Waiting for error group managing segments... Dec 28 17:50:25.192650189 (554h21m25.063733254s) [noise=true , address=145.79.227.38:4979 , piece_id=FO67DAJVGNS4YXTHGRUCALI7T32PHBFZYHRFL35TAN53VT2HDTTA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.192883012 (554h21m25.063966057s) [noise=true , address=145.79.227.36:4518 , piece_id=EJMLGB6YIGOOTKFK73GYMMPVPZDCWSVWP7IIIJ4T6MAWJFL3XZVA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.196493006 (554h21m25.067576051s) [noise=true , address=145.79.227.7:4295 , piece_id=IL2ZNGGEO7SQHYNL3O26XAZH46GY5QZVY67VQ2TQNFFHF5UOZQBA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.199101032 (554h21m25.070184077s) [noise=true , address=145.79.227.15:4874 , piece_id=NH36TESII4A6SK3SUZSD4BNMUUE6WQZDZBCJYWBRDGFG55BTTAFA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.203404667 (554h21m25.074487712s) [noise=true , address=145.79.227.41:4086 , piece_id=4TMCGC5A5HDOVGI6HPYKEWXBNYEW76A4M24XRAEIFIQWPRSK4VLA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.204280244 (554h21m25.075363279s) [noise=true , address=145.79.227.48:4303 , piece_id=HES7TTXHVQHVM5OGZH23HZGSAFJDQGY6NV7RBPRCKKWE535YLLGQ, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.204862874 (554h21m25.075945909s) [noise=true , address=145.79.227.5:4223 , piece_id=W7PHJEJAEBTPV4HRDL63FUSHND5TCLJKHOGDOR3VWWWC2JDURVEQ, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.205321758 (554h21m25.076404793s) [noise=true , address=145.79.227.26:4654 , piece_id=NCSFRKRKWGBJVDBMIUJXERBGLHANUBBCSFEXGAAUNWTGKCKY7PMA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.210965865 (554h21m25.08204891s) [noise=true , address=145.79.227.9:4366 , piece_id=IGG66AYDEXFFL6CTXCX54WTM2D4VTQYK4L7CXUIIHZU6ZVXARDIA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.212576802 (554h21m25.083659847s) [noise=true , address=145.79.227.4:4185 , piece_id=U5TWRN4VEWNCAJSKBCENXBOW3VB6MEYXPQLG65OCR6G2HDOQRI3Q, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.239354202 (554h21m25.110437247s) [noise=true , address=145.79.227.47:4265 , piece_id=DCTC4E7PKYB7MFVQZF3CWIY2J7CW7VJSYKD57TA6KKOVV7MCR63A, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.439907230 (554h21m25.310990275s) [noise=true , address=77.90.31.19:4841 , piece_id=IDEHD3OONBEK4CLFANB4S74TIXBTMOX2M7PWF3EPAQZD5IAKHYAA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.486575725 (554h21m25.35765877s) [noise=true , address=45.91.116.6:4974 , piece_id=CTU2J7R4ESM5XSU6VT5NHCICKLEC6SU4SAHWV3MUFPYOQYZPRILA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.486600252 (554h21m25.357683287s) [seg_pos={0 0}, upload=486017]: Segment reached optimal threshold of 13 pieces. Dec 28 17:50:25.486899030 (554h21m25.357982076s) [noise=true , address=77.90.31.15:4697 , piece_id=VTFX4PUEB75FODWO2BEJIZMU2YCWEEXSGNYL5AL5KDBNKDWJD4GA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 17:50:25.486907697 (554h21m25.357990742s) [noise=true , address=77.90.31.15:4697 , piece_id=VTFX4PUEB75FODWO2BEJIZMU2YCWEEXSGNYL5AL5KDBNKDWJD4GA, seg_pos={0 0}, upload=486017]: Long tail cancellation due to optimal threshold Dec 28 17:50:25.487155950 (554h21m25.358238985s) [noise=true , address=145.79.227.34:4907 , piece_id=ZIBHRDM442UFQN6XC7BKLATZ4LTQOHYDZF53BZH7PTRYZVFCGD6Q, seg_pos={0 0}, upload=486017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 17:50:25.487162262 (554h21m25.358245297s) [noise=true , address=145.79.227.34:4907 , piece_id=ZIBHRDM442UFQN6XC7BKLATZ4LTQOHYDZF53BZH7PTRYZVFCGD6Q, seg_pos={0 0}, upload=486017]: Long tail cancellation due to optimal threshold Dec 28 17:50:25.487273043 (554h21m25.358356078s) [noise=true , address=77.90.31.10:4518 , piece_id=JCYIENVACJ7WD7JPAOF7V7OTMZZ73MUOZ77RLTZVWHGD7PF6YG6A, seg_pos={0 0}, upload=486017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 28 17:50:25.487278313 (554h21m25.358361348s) [noise=true , address=77.90.31.10:4518 , piece_id=JCYIENVACJ7WD7JPAOF7V7OTMZZ73MUOZ77RLTZVWHGD7PF6YG6A, seg_pos={0 0}, upload=486017]: Long tail cancellation due to optimal threshold Dec 28 17:50:25.487294324 (554h21m25.358377369s) [seg_pos={0 0}, upload=486017]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:50:25.487365940 (554h21m25.358448985s) [upload=486017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:50:25.487371471 (554h21m25.358454516s) [upload=486017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:50:25.775734313 (554h21m25.646817359s) [upload=486017]: Done uploading segments... Dec 28 17:50:25.775760504 (554h21m25.646843549s) [upload=486017]: Upload finished. err: <nil> Duration: 6.197962261s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X..X.............X......................] duration: 3.767106209s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:51:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.767134102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [......XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........X..XX.X] duration: 30.002509388s trace: https://jaeger.dev.storj.io/trace/45b2db7853d34e40 FAIL (2025-12-28 17:50:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: CTU2J7R4ESM5XSU6VT5NHCICKLEC6SU4SAHWV3MUFPYOQYZPRILA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: IDEHD3OONBEK4CLFANB4S74TIXBTMOX2M7PWF3EPAQZD5IAKHYAA) context deadline exceeded ======LOGS====== Dec 28 17:50:22.937429599 (554h21m22.808512634s) [upload=486017]: Starting upload Dec 28 17:50:22.937504411 (554h21m22.808587456s) [upload=486017]: Done starting upload Dec 28 17:50:22.937676038 (554h21m22.808759083s) [upload=486017]: Uploading segments... Dec 28 17:50:22.937734419 (554h21m22.808817464s) [upload=486017]: Got next segment. Inline: false Dec 28 17:50:22.937766320 (554h21m22.808849355s) [upload=486017]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:50:22.937771109 (554h21m22.808854144s) [upload=486017]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:50:23.183050895 (554h21m23.05413396s) [seg_pos={0 0}, upload=486017]: Begin upload segment... Dec 28 17:50:23.183103265 (554h21m23.0541863s) [seg_pos={0 0}, upload=486017]: Done begin upload segment. Dec 28 17:50:23.183142199 (554h21m23.054225234s) [noise=true , address=145.79.227.7:4295 , piece_id=IL2ZNGGEO7SQHYNL3O26XAZH46GY5QZVY67VQ2TQNFFHF5UOZQBA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.183233053 (554h21m23.054316088s) [noise=true , address=145.79.227.15:4874 , piece_id=NH36TESII4A6SK3SUZSD4BNMUUE6WQZDZBCJYWBRDGFG55BTTAFA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.183845408 (554h21m23.054928453s) [noise=true , address=145.79.227.34:4907 , piece_id=ZIBHRDM442UFQN6XC7BKLATZ4LTQOHYDZF53BZH7PTRYZVFCGD6Q, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.184310444 (554h21m23.055393479s) [noise=true , address=145.79.227.9:4366 , piece_id=IGG66AYDEXFFL6CTXCX54WTM2D4VTQYK4L7CXUIIHZU6ZVXARDIA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.184793603 (554h21m23.055876648s) [noise=true , address=77.90.31.10:4518 , piece_id=JCYIENVACJ7WD7JPAOF7V7OTMZZ73MUOZ77RLTZVWHGD7PF6YG6A, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.185256886 (554h21m23.056339931s) [noise=true , address=145.79.227.38:4979 , piece_id=FO67DAJVGNS4YXTHGRUCALI7T32PHBFZYHRFL35TAN53VT2HDTTA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.185673670 (554h21m23.056756715s) [noise=true , address=145.79.227.36:4518 , piece_id=EJMLGB6YIGOOTKFK73GYMMPVPZDCWSVWP7IIIJ4T6MAWJFL3XZVA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.186225391 (554h21m23.057308426s) [noise=true , address=77.90.31.15:4697 , piece_id=VTFX4PUEB75FODWO2BEJIZMU2YCWEEXSGNYL5AL5KDBNKDWJD4GA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.187044229 (554h21m23.058127264s) [noise=true , address=145.79.227.47:4265 , piece_id=DCTC4E7PKYB7MFVQZF3CWIY2J7CW7VJSYKD57TA6KKOVV7MCR63A, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.188220469 (554h21m23.059303514s) [noise=true , address=45.91.116.6:4974 , piece_id=CTU2J7R4ESM5XSU6VT5NHCICKLEC6SU4SAHWV3MUFPYOQYZPRILA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.190587236 (554h21m23.061670281s) [noise=true , address=145.79.227.5:4223 , piece_id=W7PHJEJAEBTPV4HRDL63FUSHND5TCLJKHOGDOR3VWWWC2JDURVEQ, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.192530416 (554h21m23.063613451s) [noise=true , address=145.79.227.26:4654 , piece_id=NCSFRKRKWGBJVDBMIUJXERBGLHANUBBCSFEXGAAUNWTGKCKY7PMA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.193893822 (554h21m23.064976867s) [noise=true , address=145.79.227.48:4303 , piece_id=HES7TTXHVQHVM5OGZH23HZGSAFJDQGY6NV7RBPRCKKWE535YLLGQ, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.195454204 (554h21m23.066537249s) [noise=true , address=77.90.31.19:4841 , piece_id=IDEHD3OONBEK4CLFANB4S74TIXBTMOX2M7PWF3EPAQZD5IAKHYAA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.196706198 (554h21m23.067789233s) [noise=true , address=145.79.227.4:4185 , piece_id=U5TWRN4VEWNCAJSKBCENXBOW3VB6MEYXPQLG65OCR6G2HDOQRI3Q, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:23.197426038 (554h21m23.068509083s) [noise=true , address=145.79.227.41:4086 , piece_id=4TMCGC5A5HDOVGI6HPYKEWXBNYEW76A4M24XRAEIFIQWPRSK4VLA, seg_pos={0 0}, upload=486017]: Uploading piece... Dec 28 17:50:24.855859198 (554h21m24.726942244s) [upload=486017]: Next returned nil segment Dec 28 17:50:24.855879107 (554h21m24.726962152s) [upload=486017]: Waiting for error group managing segments... Dec 28 17:50:25.192650189 (554h21m25.063733254s) [noise=true , address=145.79.227.38:4979 , piece_id=FO67DAJVGNS4YXTHGRUCALI7T32PHBFZYHRFL35TAN53VT2HDTTA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.192883012 (554h21m25.063966057s) [noise=true , address=145.79.227.36:4518 , piece_id=EJMLGB6YIGOOTKFK73GYMMPVPZDCWSVWP7IIIJ4T6MAWJFL3XZVA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.196493006 (554h21m25.067576051s) [noise=true , address=145.79.227.7:4295 , piece_id=IL2ZNGGEO7SQHYNL3O26XAZH46GY5QZVY67VQ2TQNFFHF5UOZQBA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.199101032 (554h21m25.070184077s) [noise=true , address=145.79.227.15:4874 , piece_id=NH36TESII4A6SK3SUZSD4BNMUUE6WQZDZBCJYWBRDGFG55BTTAFA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.203404667 (554h21m25.074487712s) [noise=true , address=145.79.227.41:4086 , piece_id=4TMCGC5A5HDOVGI6HPYKEWXBNYEW76A4M24XRAEIFIQWPRSK4VLA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.204280244 (554h21m25.075363279s) [noise=true , address=145.79.227.48:4303 , piece_id=HES7TTXHVQHVM5OGZH23HZGSAFJDQGY6NV7RBPRCKKWE535YLLGQ, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.204862874 (554h21m25.075945909s) [noise=true , address=145.79.227.5:4223 , piece_id=W7PHJEJAEBTPV4HRDL63FUSHND5TCLJKHOGDOR3VWWWC2JDURVEQ, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.205321758 (554h21m25.076404793s) [noise=true , address=145.79.227.26:4654 , piece_id=NCSFRKRKWGBJVDBMIUJXERBGLHANUBBCSFEXGAAUNWTGKCKY7PMA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.210965865 (554h21m25.08204891s) [noise=true , address=145.79.227.9:4366 , piece_id=IGG66AYDEXFFL6CTXCX54WTM2D4VTQYK4L7CXUIIHZU6ZVXARDIA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.212576802 (554h21m25.083659847s) [noise=true , address=145.79.227.4:4185 , piece_id=U5TWRN4VEWNCAJSKBCENXBOW3VB6MEYXPQLG65OCR6G2HDOQRI3Q, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.239354202 (554h21m25.110437247s) [noise=true , address=145.79.227.47:4265 , piece_id=DCTC4E7PKYB7MFVQZF3CWIY2J7CW7VJSYKD57TA6KKOVV7MCR63A, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.439907230 (554h21m25.310990275s) [noise=true , address=77.90.31.19:4841 , piece_id=IDEHD3OONBEK4CLFANB4S74TIXBTMOX2M7PWF3EPAQZD5IAKHYAA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.486575725 (554h21m25.35765877s) [noise=true , address=45.91.116.6:4974 , piece_id=CTU2J7R4ESM5XSU6VT5NHCICKLEC6SU4SAHWV3MUFPYOQYZPRILA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: <nil> Dec 28 17:50:25.486600252 (554h21m25.357683287s) [seg_pos={0 0}, upload=486017]: Segment reached optimal threshold of 13 pieces. Dec 28 17:50:25.486899030 (554h21m25.357982076s) [noise=true , address=77.90.31.15:4697 , piece_id=VTFX4PUEB75FODWO2BEJIZMU2YCWEEXSGNYL5AL5KDBNKDWJD4GA, seg_pos={0 0}, upload=486017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 17:50:25.486907697 (554h21m25.357990742s) [noise=true , address=77.90.31.15:4697 , piece_id=VTFX4PUEB75FODWO2BEJIZMU2YCWEEXSGNYL5AL5KDBNKDWJD4GA, seg_pos={0 0}, upload=486017]: Long tail cancellation due to optimal threshold Dec 28 17:50:25.487155950 (554h21m25.358238985s) [noise=true , address=145.79.227.34:4907 , piece_id=ZIBHRDM442UFQN6XC7BKLATZ4LTQOHYDZF53BZH7PTRYZVFCGD6Q, seg_pos={0 0}, upload=486017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 17:50:25.487162262 (554h21m25.358245297s) [noise=true , address=145.79.227.34:4907 , piece_id=ZIBHRDM442UFQN6XC7BKLATZ4LTQOHYDZF53BZH7PTRYZVFCGD6Q, seg_pos={0 0}, upload=486017]: Long tail cancellation due to optimal threshold Dec 28 17:50:25.487273043 (554h21m25.358356078s) [noise=true , address=77.90.31.10:4518 , piece_id=JCYIENVACJ7WD7JPAOF7V7OTMZZ73MUOZ77RLTZVWHGD7PF6YG6A, seg_pos={0 0}, upload=486017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 28 17:50:25.487278313 (554h21m25.358361348s) [noise=true , address=77.90.31.10:4518 , piece_id=JCYIENVACJ7WD7JPAOF7V7OTMZZ73MUOZ77RLTZVWHGD7PF6YG6A, seg_pos={0 0}, upload=486017]: Long tail cancellation due to optimal threshold Dec 28 17:50:25.487294324 (554h21m25.358377369s) [seg_pos={0 0}, upload=486017]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:50:25.487365940 (554h21m25.358448985s) [upload=486017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:50:25.487371471 (554h21m25.358454516s) [upload=486017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:50:25.775734313 (554h21m25.646817359s) [upload=486017]: Done uploading segments... Dec 28 17:50:25.775760504 (554h21m25.646843549s) [upload=486017]: Upload finished. err: <nil> Duration: 30.002534425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 38.019350255s trace: https://jaeger.dev.storj.io/trace/3d8b79377b19fbe5 FAIL (2025-12-28 17:50:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gin upload segment. Dec 28 17:44:30.976730747 (554h15m30.847813782s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.976740326 (554h15m30.847823361s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977238735 (554h15m30.84832177s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977395152 (554h15m30.848478187s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.979914119 (554h15m30.850997164s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.980878776 (554h15m30.851961811s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.981975264 (554h15m30.853058309s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982279864 (554h15m30.853362899s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982969287 (554h15m30.854052332s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.983667216 (554h15m30.854750271s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.984125969 (554h15m30.855209014s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985053625 (554h15m30.85613667s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985637347 (554h15m30.856720382s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985740974 (554h15m30.856824039s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.989521163 (554h15m30.860604198s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.990672796 (554h15m30.861755831s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.991767229 (554h15m30.862850284s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.993443933 (554h15m30.864526978s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:31.878310708 (554h15m31.749393753s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: ecclient: upload failed (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, address:193.149.164.143:4694): protocol: storage node overloaded, request limit: 2000 Dec 28 17:44:32.439383612 (554h15m32.310466647s) [upload=485932]: Next returned nil segment Dec 28 17:44:32.439397199 (554h15m32.310480234s) [upload=485932]: Waiting for error group managing segments... Dec 28 17:44:32.845179451 (554h15m32.716262486s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857045366 (554h15m32.728128401s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857359254 (554h15m32.728442289s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.907044634 (554h15m32.778127669s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.062870003 (554h15m32.933953048s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.152338938 (554h15m33.023421983s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.156658764 (554h15m33.027741799s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.196623976 (554h15m33.067707011s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.726903934 (554h15m33.597986979s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.746518434 (554h15m33.617601469s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.774407711 (554h15m33.645490756s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.823214066 (554h15m33.694297121s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.887815218 (554h15m33.758898253s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895889844 (554h15m33.766972879s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895905184 (554h15m33.766988229s) [seg_pos={0 0}, upload=485932]: Segment reached optimal threshold of 14 pieces. Dec 28 17:44:33.896119602 (554h15m33.767202637s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896129150 (554h15m33.767212205s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896347405 (554h15m33.76743045s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896385608 (554h15m33.767468653s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896453226 (554h15m33.767536261s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896465571 (554h15m33.767548626s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896487211 (554h15m33.767570246s) [seg_pos={0 0}, upload=485932]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:44:33.896575890 (554h15m33.767658936s) [upload=485932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:44:33.896582042 (554h15m33.767665087s) [upload=485932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:44:34.190733098 (554h15m34.061816143s) [upload=485932]: Done uploading segments... Dec 28 17:44:34.190758908 (554h15m34.061841943s) [upload=485932]: Upload finished. err: <nil> Duration: 38.0193894s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X....................X..X.............X.....................] duration: 3.437557388s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:50:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.437585281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 26.275293708s trace: https://jaeger.dev.storj.io/trace/3d8b79377b19fbe5 FAIL (2025-12-28 17:50:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gin upload segment. Dec 28 17:44:30.976730747 (554h15m30.847813782s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.976740326 (554h15m30.847823361s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977238735 (554h15m30.84832177s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977395152 (554h15m30.848478187s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.979914119 (554h15m30.850997164s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.980878776 (554h15m30.851961811s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.981975264 (554h15m30.853058309s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982279864 (554h15m30.853362899s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982969287 (554h15m30.854052332s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.983667216 (554h15m30.854750271s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.984125969 (554h15m30.855209014s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985053625 (554h15m30.85613667s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985637347 (554h15m30.856720382s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985740974 (554h15m30.856824039s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.989521163 (554h15m30.860604198s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.990672796 (554h15m30.861755831s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.991767229 (554h15m30.862850284s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.993443933 (554h15m30.864526978s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:31.878310708 (554h15m31.749393753s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: ecclient: upload failed (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, address:193.149.164.143:4694): protocol: storage node overloaded, request limit: 2000 Dec 28 17:44:32.439383612 (554h15m32.310466647s) [upload=485932]: Next returned nil segment Dec 28 17:44:32.439397199 (554h15m32.310480234s) [upload=485932]: Waiting for error group managing segments... Dec 28 17:44:32.845179451 (554h15m32.716262486s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857045366 (554h15m32.728128401s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857359254 (554h15m32.728442289s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.907044634 (554h15m32.778127669s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.062870003 (554h15m32.933953048s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.152338938 (554h15m33.023421983s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.156658764 (554h15m33.027741799s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.196623976 (554h15m33.067707011s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.726903934 (554h15m33.597986979s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.746518434 (554h15m33.617601469s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.774407711 (554h15m33.645490756s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.823214066 (554h15m33.694297121s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.887815218 (554h15m33.758898253s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895889844 (554h15m33.766972879s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895905184 (554h15m33.766988229s) [seg_pos={0 0}, upload=485932]: Segment reached optimal threshold of 14 pieces. Dec 28 17:44:33.896119602 (554h15m33.767202637s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896129150 (554h15m33.767212205s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896347405 (554h15m33.76743045s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896385608 (554h15m33.767468653s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896453226 (554h15m33.767536261s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896465571 (554h15m33.767548626s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896487211 (554h15m33.767570246s) [seg_pos={0 0}, upload=485932]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:44:33.896575890 (554h15m33.767658936s) [upload=485932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:44:33.896582042 (554h15m33.767665087s) [upload=485932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:44:34.190733098 (554h15m34.061816143s) [upload=485932]: Done uploading segments... Dec 28 17:44:34.190758908 (554h15m34.061841943s) [upload=485932]: Upload finished. err: <nil> Duration: 26.275324188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.......XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........X..XX.] duration: 8.105599038s trace: https://jaeger.dev.storj.io/trace/12e9797b7a3bb81b FAIL (2025-12-28 17:49:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: J2467DEXCEHGUXYCXYRHWT5K4PWFKOXJLSFKGNGEVZCPJ2GJJGYA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: EVMGEJXCVDY4KZHYWCDCVAGPIXIV6O4KKPSCD6APCFFNLDNINQEA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QKKILXTJNHFJ57HVF3DMVIXG5UURKZRGNZ36CMYQTNE2W5FNOOEQ) context deadline exceeded ======LOGS====== Dec 28 17:48:47.491664695 (554h19m47.36274775s) [upload=485995]: Starting upload Dec 28 17:48:47.491795064 (554h19m47.362878109s) [upload=485995]: Done starting upload Dec 28 17:48:47.491894092 (554h19m47.362977137s) [upload=485995]: Uploading segments... Dec 28 17:48:47.492092260 (554h19m47.363175305s) [upload=485995]: Got next segment. Inline: false Dec 28 17:48:47.492213300 (554h19m47.363296335s) [upload=485995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:48:47.492220834 (554h19m47.363303879s) [upload=485995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:48:47.741207158 (554h19m47.612290204s) [seg_pos={0 0}, upload=485995]: Begin upload segment... Dec 28 17:48:47.741265289 (554h19m47.612348325s) [seg_pos={0 0}, upload=485995]: Done begin upload segment. Dec 28 17:48:47.741283033 (554h19m47.612366078s) [noise=true , address=145.79.227.17:4971 , piece_id=PFE2SDPCNWADR2N7HCBHEZXLTTBFZMW7Z4ACBGJO45KOKOOGUXUQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.741320855 (554h19m47.61240389s) [noise=true , address=45.91.116.7:4011 , piece_id=EVMGEJXCVDY4KZHYWCDCVAGPIXIV6O4KKPSCD6APCFFNLDNINQEA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.742122351 (554h19m47.613205396s) [noise=true , address=145.79.227.15:4875 , piece_id=F64QTOUSW72L4Q6S7PPAELRJNVZMF2OVNQUKTTFDZGVIUCGCBJXQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.742534546 (554h19m47.613617581s) [noise=true , address=145.79.227.11:4729 , piece_id=YNRHLP6XSSW2CWDKAYVNKNB7AZ4EGXJHMH6V6O2RR3C67JWSN22Q, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.742918928 (554h19m47.614001963s) [noise=true , address=45.91.116.8:4045 , piece_id=WPIB43OGNLQOBYH4IPVDPOUK2FQYXCKPBXH5ISQD6FOTZAVJ6ACQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.743724982 (554h19m47.614808018s) [noise=true , address=77.90.31.18:4805 , piece_id=QKKILXTJNHFJ57HVF3DMVIXG5UURKZRGNZ36CMYQTNE2W5FNOOEQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.744689179 (554h19m47.615772214s) [noise=true , address=45.91.116.14:4191 , piece_id=S5ZWMVB6DMDKVIXKX2VNQITA2LZ6OFHKMH646XPMKTC3RTLMW2XQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.746268155 (554h19m47.6173512s) [noise=true , address=145.79.227.26:4654 , piece_id=R2M4CP4ZF45WEBUFGVD7LD6YYGU5PXTS37EQQUE2EKOPKV6MFZ4Q, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.746434001 (554h19m47.617517046s) [noise=true , address=145.79.227.52:4445 , piece_id=AU24JSJ7HZ6BWIAETX4ABGE46HYSWFH4DW56BCTFN25SY4SK4L2A, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.748407860 (554h19m47.619490905s) [noise=true , address=45.91.116.10:4084 , piece_id=J2467DEXCEHGUXYCXYRHWT5K4PWFKOXJLSFKGNGEVZCPJ2GJJGYA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.750412497 (554h19m47.621495542s) [noise=true , address=77.90.31.10:4518 , piece_id=73MIWUJDFZ6LZMONRLXS3IN5VSN6YEHAR5HWXTSLHINISRMOJGUA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.752069612 (554h19m47.623152657s) [noise=true , address=145.79.227.42:4121 , piece_id=NSPNBETEE5WI3GTMU5RHBOCT5EIBC24VVXDMIGBB467AHD2I5IRA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.753914335 (554h19m47.62499739s) [noise=true , address=45.91.116.11:4117 , piece_id=OFVDYII3E5BVIOLDREPX76MMZPL2N3P7CVPFHPDK2JMIHMLFDNPQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.755179624 (554h19m47.626262669s) [noise=true , address=145.79.227.50:4374 , piece_id=Q6RJ63BJIOXTDTZKSZDYWEJ5BDU3V6HITHAY25Z3QWWDNUPQ6HHA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.756953251 (554h19m47.628036286s) [noise=true , address=45.91.116.13:4154 , piece_id=6M5EWQMWZVKEH6FPWZM5JU2KML63MSPA7WMZRAOW66YRMRBHXKEA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.757305873 (554h19m47.628388918s) [noise=true , address=145.79.227.6:4259 , piece_id=QDTU5M64JWRGVCSNL5ADSPKW4BKOFSY46FLYLA7FSHLKSS7TN7TQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:49.446191802 (554h19m49.317274847s) [upload=485995]: Next returned nil segment Dec 28 17:48:49.446208413 (554h19m49.317291448s) [upload=485995]: Waiting for error group managing segments... Dec 28 17:48:49.749571600 (554h19m49.620654655s) [noise=true , address=145.79.227.17:4971 , piece_id=PFE2SDPCNWADR2N7HCBHEZXLTTBFZMW7Z4ACBGJO45KOKOOGUXUQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.752623853 (554h19m49.623706888s) [noise=true , address=145.79.227.26:4654 , piece_id=R2M4CP4ZF45WEBUFGVD7LD6YYGU5PXTS37EQQUE2EKOPKV6MFZ4Q, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.754907460 (554h19m49.625990505s) [noise=true , address=145.79.227.52:4445 , piece_id=AU24JSJ7HZ6BWIAETX4ABGE46HYSWFH4DW56BCTFN25SY4SK4L2A, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.757474849 (554h19m49.628557894s) [noise=true , address=145.79.227.42:4121 , piece_id=NSPNBETEE5WI3GTMU5RHBOCT5EIBC24VVXDMIGBB467AHD2I5IRA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.768279624 (554h19m49.639362669s) [noise=true , address=145.79.227.50:4374 , piece_id=Q6RJ63BJIOXTDTZKSZDYWEJ5BDU3V6HITHAY25Z3QWWDNUPQ6HHA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.775414821 (554h19m49.646497856s) [noise=true , address=145.79.227.6:4259 , piece_id=QDTU5M64JWRGVCSNL5ADSPKW4BKOFSY46FLYLA7FSHLKSS7TN7TQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.775498660 (554h19m49.646581705s) [noise=true , address=145.79.227.15:4875 , piece_id=F64QTOUSW72L4Q6S7PPAELRJNVZMF2OVNQUKTTFDZGVIUCGCBJXQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.782638817 (554h19m49.653721862s) [noise=true , address=145.79.227.11:4729 , piece_id=YNRHLP6XSSW2CWDKAYVNKNB7AZ4EGXJHMH6V6O2RR3C67JWSN22Q, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.935803339 (554h19m49.806886374s) [noise=true , address=77.90.31.10:4518 , piece_id=73MIWUJDFZ6LZMONRLXS3IN5VSN6YEHAR5HWXTSLHINISRMOJGUA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.970562597 (554h19m49.841645642s) [noise=true , address=45.91.116.14:4191 , piece_id=S5ZWMVB6DMDKVIXKX2VNQITA2LZ6OFHKMH646XPMKTC3RTLMW2XQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:50.003925886 (554h19m49.875008921s) [noise=true , address=45.91.116.10:4084 , piece_id=J2467DEXCEHGUXYCXYRHWT5K4PWFKOXJLSFKGNGEVZCPJ2GJJGYA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:50.048997741 (554h19m49.920080776s) [noise=true , address=77.90.31.18:4805 , piece_id=QKKILXTJNHFJ57HVF3DMVIXG5UURKZRGNZ36CMYQTNE2W5FNOOEQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:50.077950702 (554h19m49.949033737s) [noise=true , address=45.91.116.7:4011 , piece_id=EVMGEJXCVDY4KZHYWCDCVAGPIXIV6O4KKPSCD6APCFFNLDNINQEA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:50.077978966 (554h19m49.949062012s) [seg_pos={0 0}, upload=485995]: Segment reached optimal threshold of 13 pieces. Dec 28 17:48:50.078208874 (554h19m49.949291919s) [noise=true , address=45.91.116.8:4045 , piece_id=WPIB43OGNLQOBYH4IPVDPOUK2FQYXCKPBXH5ISQD6FOTZAVJ6ACQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:49716->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:49716->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49716->45.91.116.8:4045: use of closed network connection Dec 28 17:48:50.078217932 (554h19m49.949300977s) [noise=true , address=45.91.116.8:4045 , piece_id=WPIB43OGNLQOBYH4IPVDPOUK2FQYXCKPBXH5ISQD6FOTZAVJ6ACQ, seg_pos={0 0}, upload=485995]: Long tail cancellation due to optimal threshold Dec 28 17:48:50.078246515 (554h19m49.94932955s) [noise=true , address=45.91.116.11:4117 , piece_id=OFVDYII3E5BVIOLDREPX76MMZPL2N3P7CVPFHPDK2JMIHMLFDNPQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 28 17:48:50.078274378 (554h19m49.949357424s) [noise=true , address=45.91.116.11:4117 , piece_id=OFVDYII3E5BVIOLDREPX76MMZPL2N3P7CVPFHPDK2JMIHMLFDNPQ, seg_pos={0 0}, upload=485995]: Long tail cancellation due to optimal threshold Dec 28 17:48:50.078625216 (554h19m49.949708252s) [noise=true , address=45.91.116.13:4154 , piece_id=6M5EWQMWZVKEH6FPWZM5JU2KML63MSPA7WMZRAOW66YRMRBHXKEA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 17:48:50.078634794 (554h19m49.949717839s) [noise=true , address=45.91.116.13:4154 , piece_id=6M5EWQMWZVKEH6FPWZM5JU2KML63MSPA7WMZRAOW66YRMRBHXKEA, seg_pos={0 0}, upload=485995]: Long tail cancellation due to optimal threshold Dec 28 17:48:50.078654733 (554h19m49.949737778s) [seg_pos={0 0}, upload=485995]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:48:50.078781424 (554h19m49.949864459s) [upload=485995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:48:50.078786805 (554h19m49.94986985s) [upload=485995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:48:50.370267566 (554h19m50.241350601s) [upload=485995]: Done uploading segments... Dec 28 17:48:50.370285419 (554h19m50.241368454s) [upload=485995]: Upload finished. err: <nil> Duration: 8.105614898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [........XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........X..XX] duration: 30.004482964s trace: https://jaeger.dev.storj.io/trace/12e9797b7a3bb81b FAIL (2025-12-28 17:49:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: J2467DEXCEHGUXYCXYRHWT5K4PWFKOXJLSFKGNGEVZCPJ2GJJGYA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: EVMGEJXCVDY4KZHYWCDCVAGPIXIV6O4KKPSCD6APCFFNLDNINQEA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: QKKILXTJNHFJ57HVF3DMVIXG5UURKZRGNZ36CMYQTNE2W5FNOOEQ) context deadline exceeded ======LOGS====== Dec 28 17:48:47.491664695 (554h19m47.36274775s) [upload=485995]: Starting upload Dec 28 17:48:47.491795064 (554h19m47.362878109s) [upload=485995]: Done starting upload Dec 28 17:48:47.491894092 (554h19m47.362977137s) [upload=485995]: Uploading segments... Dec 28 17:48:47.492092260 (554h19m47.363175305s) [upload=485995]: Got next segment. Inline: false Dec 28 17:48:47.492213300 (554h19m47.363296335s) [upload=485995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:48:47.492220834 (554h19m47.363303879s) [upload=485995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:48:47.741207158 (554h19m47.612290204s) [seg_pos={0 0}, upload=485995]: Begin upload segment... Dec 28 17:48:47.741265289 (554h19m47.612348325s) [seg_pos={0 0}, upload=485995]: Done begin upload segment. Dec 28 17:48:47.741283033 (554h19m47.612366078s) [noise=true , address=145.79.227.17:4971 , piece_id=PFE2SDPCNWADR2N7HCBHEZXLTTBFZMW7Z4ACBGJO45KOKOOGUXUQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.741320855 (554h19m47.61240389s) [noise=true , address=45.91.116.7:4011 , piece_id=EVMGEJXCVDY4KZHYWCDCVAGPIXIV6O4KKPSCD6APCFFNLDNINQEA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.742122351 (554h19m47.613205396s) [noise=true , address=145.79.227.15:4875 , piece_id=F64QTOUSW72L4Q6S7PPAELRJNVZMF2OVNQUKTTFDZGVIUCGCBJXQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.742534546 (554h19m47.613617581s) [noise=true , address=145.79.227.11:4729 , piece_id=YNRHLP6XSSW2CWDKAYVNKNB7AZ4EGXJHMH6V6O2RR3C67JWSN22Q, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.742918928 (554h19m47.614001963s) [noise=true , address=45.91.116.8:4045 , piece_id=WPIB43OGNLQOBYH4IPVDPOUK2FQYXCKPBXH5ISQD6FOTZAVJ6ACQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.743724982 (554h19m47.614808018s) [noise=true , address=77.90.31.18:4805 , piece_id=QKKILXTJNHFJ57HVF3DMVIXG5UURKZRGNZ36CMYQTNE2W5FNOOEQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.744689179 (554h19m47.615772214s) [noise=true , address=45.91.116.14:4191 , piece_id=S5ZWMVB6DMDKVIXKX2VNQITA2LZ6OFHKMH646XPMKTC3RTLMW2XQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.746268155 (554h19m47.6173512s) [noise=true , address=145.79.227.26:4654 , piece_id=R2M4CP4ZF45WEBUFGVD7LD6YYGU5PXTS37EQQUE2EKOPKV6MFZ4Q, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.746434001 (554h19m47.617517046s) [noise=true , address=145.79.227.52:4445 , piece_id=AU24JSJ7HZ6BWIAETX4ABGE46HYSWFH4DW56BCTFN25SY4SK4L2A, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.748407860 (554h19m47.619490905s) [noise=true , address=45.91.116.10:4084 , piece_id=J2467DEXCEHGUXYCXYRHWT5K4PWFKOXJLSFKGNGEVZCPJ2GJJGYA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.750412497 (554h19m47.621495542s) [noise=true , address=77.90.31.10:4518 , piece_id=73MIWUJDFZ6LZMONRLXS3IN5VSN6YEHAR5HWXTSLHINISRMOJGUA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.752069612 (554h19m47.623152657s) [noise=true , address=145.79.227.42:4121 , piece_id=NSPNBETEE5WI3GTMU5RHBOCT5EIBC24VVXDMIGBB467AHD2I5IRA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.753914335 (554h19m47.62499739s) [noise=true , address=45.91.116.11:4117 , piece_id=OFVDYII3E5BVIOLDREPX76MMZPL2N3P7CVPFHPDK2JMIHMLFDNPQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.755179624 (554h19m47.626262669s) [noise=true , address=145.79.227.50:4374 , piece_id=Q6RJ63BJIOXTDTZKSZDYWEJ5BDU3V6HITHAY25Z3QWWDNUPQ6HHA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.756953251 (554h19m47.628036286s) [noise=true , address=45.91.116.13:4154 , piece_id=6M5EWQMWZVKEH6FPWZM5JU2KML63MSPA7WMZRAOW66YRMRBHXKEA, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:47.757305873 (554h19m47.628388918s) [noise=true , address=145.79.227.6:4259 , piece_id=QDTU5M64JWRGVCSNL5ADSPKW4BKOFSY46FLYLA7FSHLKSS7TN7TQ, seg_pos={0 0}, upload=485995]: Uploading piece... Dec 28 17:48:49.446191802 (554h19m49.317274847s) [upload=485995]: Next returned nil segment Dec 28 17:48:49.446208413 (554h19m49.317291448s) [upload=485995]: Waiting for error group managing segments... Dec 28 17:48:49.749571600 (554h19m49.620654655s) [noise=true , address=145.79.227.17:4971 , piece_id=PFE2SDPCNWADR2N7HCBHEZXLTTBFZMW7Z4ACBGJO45KOKOOGUXUQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.752623853 (554h19m49.623706888s) [noise=true , address=145.79.227.26:4654 , piece_id=R2M4CP4ZF45WEBUFGVD7LD6YYGU5PXTS37EQQUE2EKOPKV6MFZ4Q, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.754907460 (554h19m49.625990505s) [noise=true , address=145.79.227.52:4445 , piece_id=AU24JSJ7HZ6BWIAETX4ABGE46HYSWFH4DW56BCTFN25SY4SK4L2A, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.757474849 (554h19m49.628557894s) [noise=true , address=145.79.227.42:4121 , piece_id=NSPNBETEE5WI3GTMU5RHBOCT5EIBC24VVXDMIGBB467AHD2I5IRA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.768279624 (554h19m49.639362669s) [noise=true , address=145.79.227.50:4374 , piece_id=Q6RJ63BJIOXTDTZKSZDYWEJ5BDU3V6HITHAY25Z3QWWDNUPQ6HHA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.775414821 (554h19m49.646497856s) [noise=true , address=145.79.227.6:4259 , piece_id=QDTU5M64JWRGVCSNL5ADSPKW4BKOFSY46FLYLA7FSHLKSS7TN7TQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.775498660 (554h19m49.646581705s) [noise=true , address=145.79.227.15:4875 , piece_id=F64QTOUSW72L4Q6S7PPAELRJNVZMF2OVNQUKTTFDZGVIUCGCBJXQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.782638817 (554h19m49.653721862s) [noise=true , address=145.79.227.11:4729 , piece_id=YNRHLP6XSSW2CWDKAYVNKNB7AZ4EGXJHMH6V6O2RR3C67JWSN22Q, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.935803339 (554h19m49.806886374s) [noise=true , address=77.90.31.10:4518 , piece_id=73MIWUJDFZ6LZMONRLXS3IN5VSN6YEHAR5HWXTSLHINISRMOJGUA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:49.970562597 (554h19m49.841645642s) [noise=true , address=45.91.116.14:4191 , piece_id=S5ZWMVB6DMDKVIXKX2VNQITA2LZ6OFHKMH646XPMKTC3RTLMW2XQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:50.003925886 (554h19m49.875008921s) [noise=true , address=45.91.116.10:4084 , piece_id=J2467DEXCEHGUXYCXYRHWT5K4PWFKOXJLSFKGNGEVZCPJ2GJJGYA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:50.048997741 (554h19m49.920080776s) [noise=true , address=77.90.31.18:4805 , piece_id=QKKILXTJNHFJ57HVF3DMVIXG5UURKZRGNZ36CMYQTNE2W5FNOOEQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:50.077950702 (554h19m49.949033737s) [noise=true , address=45.91.116.7:4011 , piece_id=EVMGEJXCVDY4KZHYWCDCVAGPIXIV6O4KKPSCD6APCFFNLDNINQEA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: <nil> Dec 28 17:48:50.077978966 (554h19m49.949062012s) [seg_pos={0 0}, upload=485995]: Segment reached optimal threshold of 13 pieces. Dec 28 17:48:50.078208874 (554h19m49.949291919s) [noise=true , address=45.91.116.8:4045 , piece_id=WPIB43OGNLQOBYH4IPVDPOUK2FQYXCKPBXH5ISQD6FOTZAVJ6ACQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:49716->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:49716->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49716->45.91.116.8:4045: use of closed network connection Dec 28 17:48:50.078217932 (554h19m49.949300977s) [noise=true , address=45.91.116.8:4045 , piece_id=WPIB43OGNLQOBYH4IPVDPOUK2FQYXCKPBXH5ISQD6FOTZAVJ6ACQ, seg_pos={0 0}, upload=485995]: Long tail cancellation due to optimal threshold Dec 28 17:48:50.078246515 (554h19m49.94932955s) [noise=true , address=45.91.116.11:4117 , piece_id=OFVDYII3E5BVIOLDREPX76MMZPL2N3P7CVPFHPDK2JMIHMLFDNPQ, seg_pos={0 0}, upload=485995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 28 17:48:50.078274378 (554h19m49.949357424s) [noise=true , address=45.91.116.11:4117 , piece_id=OFVDYII3E5BVIOLDREPX76MMZPL2N3P7CVPFHPDK2JMIHMLFDNPQ, seg_pos={0 0}, upload=485995]: Long tail cancellation due to optimal threshold Dec 28 17:48:50.078625216 (554h19m49.949708252s) [noise=true , address=45.91.116.13:4154 , piece_id=6M5EWQMWZVKEH6FPWZM5JU2KML63MSPA7WMZRAOW66YRMRBHXKEA, seg_pos={0 0}, upload=485995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 17:48:50.078634794 (554h19m49.949717839s) [noise=true , address=45.91.116.13:4154 , piece_id=6M5EWQMWZVKEH6FPWZM5JU2KML63MSPA7WMZRAOW66YRMRBHXKEA, seg_pos={0 0}, upload=485995]: Long tail cancellation due to optimal threshold Dec 28 17:48:50.078654733 (554h19m49.949737778s) [seg_pos={0 0}, upload=485995]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:48:50.078781424 (554h19m49.949864459s) [upload=485995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:48:50.078786805 (554h19m49.94986985s) [upload=485995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:48:50.370267566 (554h19m50.241350601s) [upload=485995]: Done uploading segments... Dec 28 17:48:50.370285419 (554h19m50.241368454s) [upload=485995]: Upload finished. err: <nil> Duration: 30.004494646s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X....................X..X.............X....................] duration: 5.001282148s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:49:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001306694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 31.87922638s trace: https://jaeger.dev.storj.io/trace/3d8b79377b19fbe5 FAIL (2025-12-28 17:48:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gin upload segment. Dec 28 17:44:30.976730747 (554h15m30.847813782s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.976740326 (554h15m30.847823361s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977238735 (554h15m30.84832177s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977395152 (554h15m30.848478187s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.979914119 (554h15m30.850997164s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.980878776 (554h15m30.851961811s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.981975264 (554h15m30.853058309s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982279864 (554h15m30.853362899s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982969287 (554h15m30.854052332s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.983667216 (554h15m30.854750271s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.984125969 (554h15m30.855209014s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985053625 (554h15m30.85613667s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985637347 (554h15m30.856720382s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985740974 (554h15m30.856824039s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.989521163 (554h15m30.860604198s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.990672796 (554h15m30.861755831s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.991767229 (554h15m30.862850284s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.993443933 (554h15m30.864526978s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:31.878310708 (554h15m31.749393753s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: ecclient: upload failed (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, address:193.149.164.143:4694): protocol: storage node overloaded, request limit: 2000 Dec 28 17:44:32.439383612 (554h15m32.310466647s) [upload=485932]: Next returned nil segment Dec 28 17:44:32.439397199 (554h15m32.310480234s) [upload=485932]: Waiting for error group managing segments... Dec 28 17:44:32.845179451 (554h15m32.716262486s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857045366 (554h15m32.728128401s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857359254 (554h15m32.728442289s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.907044634 (554h15m32.778127669s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.062870003 (554h15m32.933953048s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.152338938 (554h15m33.023421983s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.156658764 (554h15m33.027741799s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.196623976 (554h15m33.067707011s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.726903934 (554h15m33.597986979s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.746518434 (554h15m33.617601469s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.774407711 (554h15m33.645490756s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.823214066 (554h15m33.694297121s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.887815218 (554h15m33.758898253s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895889844 (554h15m33.766972879s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895905184 (554h15m33.766988229s) [seg_pos={0 0}, upload=485932]: Segment reached optimal threshold of 14 pieces. Dec 28 17:44:33.896119602 (554h15m33.767202637s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896129150 (554h15m33.767212205s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896347405 (554h15m33.76743045s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896385608 (554h15m33.767468653s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896453226 (554h15m33.767536261s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896465571 (554h15m33.767548626s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896487211 (554h15m33.767570246s) [seg_pos={0 0}, upload=485932]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:44:33.896575890 (554h15m33.767658936s) [upload=485932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:44:33.896582042 (554h15m33.767665087s) [upload=485932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:44:34.190733098 (554h15m34.061816143s) [upload=485932]: Done uploading segments... Dec 28 17:44:34.190758908 (554h15m34.061841943s) [upload=485932]: Upload finished. err: <nil> Duration: 31.879241849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.........XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........X..X] duration: 30.003469904s trace: https://jaeger.dev.storj.io/trace/4837a6c8e07402d2 FAIL (2025-12-28 17:48:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VEEZX3RB3ZXHZI6UXZLFKCQ3XLIN7QJ7OMAHZSDP4IKKHFEG32NQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YELV2XVP4LMCOVRKWFRYND4NX237E4MUTB75LB6TDTK4FT62GJYA) context deadline exceeded ======LOGS====== Dec 28 17:48:17.258032138 (554h19m17.129115183s) [upload=485986]: Starting upload Dec 28 17:48:17.258126157 (554h19m17.129209202s) [upload=485986]: Done starting upload Dec 28 17:48:17.258223462 (554h19m17.129306507s) [upload=485986]: Uploading segments... Dec 28 17:48:17.258360723 (554h19m17.129443768s) [upload=485986]: Got next segment. Inline: false Dec 28 17:48:17.258404116 (554h19m17.129487151s) [upload=485986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:48:17.258410198 (554h19m17.129493233s) [upload=485986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:48:17.504991392 (554h19m17.376074447s) [seg_pos={0 0}, upload=485986]: Begin upload segment... Dec 28 17:48:17.505058580 (554h19m17.376141615s) [seg_pos={0 0}, upload=485986]: Done begin upload segment. Dec 28 17:48:17.505110879 (554h19m17.376193924s) [noise=true , address=145.79.227.15:4873 , piece_id=LXL23F7EWSVJSTR2IXEMZXWFF2IUDA3LLZBEYPETRQWIOBILEG4A, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.505119164 (554h19m17.376202199s) [noise=true , address=145.79.227.14:4837 , piece_id=72K6MXJ5YGZWJUC6S7DQGIYDXDHYAKUXDCXJ7JRM5ARIOB5LCGOA, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.505566396 (554h19m17.376649441s) [noise=true , address=145.79.227.51:4410 , piece_id=YFUV7BGIVMN7AVAJO2CCLKO5O2AV42LHSEVO7DBYBBX4NWB2BSGA, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.506444258 (554h19m17.377527293s) [noise=true , address=145.79.227.7:4295 , piece_id=SVKMYSBU4OS3ISBDTJSTD7H2VWYRRHEIU5FUNDNPPHYUZRY6TO2A, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.506849369 (554h19m17.377932414s) [noise=true , address=145.79.227.52:4445 , piece_id=7MDJJM4AL3OYSJZHVAAWN3TCRB2H2FGQ5B6C4ECOMT6BVZFXQC7A, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.507174318 (554h19m17.378257353s) [noise=true , address=45.91.116.8:4046 , piece_id=TDUBKJZ6FE4CCQLRNA42WEVU3HUBBVNE7RZIR2ROULFQ2OEOO2IQ, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.508863234 (554h19m17.379946279s) [noise=true , address=145.79.227.46:4603 , piece_id=HGIE5UIQWQYSNNV4O72TBWG42AUSXDDH3L5J5XVHYEB5QHCLXYRA, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.509734343 (554h19m17.380817378s) [noise=true , address=145.79.227.36:4518 , piece_id=XKGUW73TGFLUNCGBQDUJQYVDACDSCNWINAOTX4R42QN7VPZKMM3A, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.510059582 (554h19m17.381142627s) [noise=true , address=45.91.116.6:4975 , piece_id=VEEZX3RB3ZXHZI6UXZLFKCQ3XLIN7QJ7OMAHZSDP4IKKHFEG32NQ, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.511163314 (554h19m17.382246349s) [noise=true , address=145.79.227.9:4366 , piece_id=R4ZMQMHZ2RMW3PFE4XGBVL3QJDTJTTCEE5H3H3T3IRAVIPM4J2NQ, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.511556172 (554h19m17.382639217s) [noise=true , address=145.79.227.47:4266 , piece_id=KOWY6SVD56MF5MCVPQN7M6I4TDIN4VEEMZYHVTUQVEDKSKDA5PZQ, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.513584794 (554h19m17.384667839s) [noise=true , address=145.79.227.41:4085 , piece_id=GLBSELA3LZHQT7LV2JFVGAPGD5BBW67MENOUKNS22RFZ2EZ6JHPA, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.514877978 (554h19m17.385961023s) [noise=true , address=45.91.116.7:4010 , piece_id=BQMP4VMMN5D5HQCWPHAHCR5RU5VCYCUS6MPPHHO6DBCO5G5WHGKQ, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.516005574 (554h19m17.387088609s) [noise=true , address=45.91.116.9:4263 , piece_id=YELV2XVP4LMCOVRKWFRYND4NX237E4MUTB75LB6TDTK4FT62GJYA, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.517472157 (554h19m17.388555192s) [noise=true , address=45.91.116.5:4297 , piece_id=IAKSE7B6THY3ZH3ZW4XBT6ASYZRVKVOR34EB4EL7O7VFKSEJWUGA, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:17.517745337 (554h19m17.388828372s) [noise=true , address=45.91.116.12:4228 , piece_id=GQIOFAID4CORB7B5MBQ756KX25WQTK6GBPRBNAJXPDUZNS2NIKSQ, seg_pos={0 0}, upload=485986]: Uploading piece... Dec 28 17:48:19.173781583 (554h19m19.044864629s) [upload=485986]: Next returned nil segment Dec 28 17:48:19.174145497 (554h19m19.045228562s) [upload=485986]: Waiting for error group managing segments... Dec 28 17:48:19.506224945 (554h19m19.37730799s) [noise=true , address=145.79.227.52:4445 , piece_id=7MDJJM4AL3OYSJZHVAAWN3TCRB2H2FGQ5B6C4ECOMT6BVZFXQC7A, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.506855455 (554h19m19.377938491s) [noise=true , address=145.79.227.46:4603 , piece_id=HGIE5UIQWQYSNNV4O72TBWG42AUSXDDH3L5J5XVHYEB5QHCLXYRA, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.514586136 (554h19m19.385669171s) [noise=true , address=145.79.227.15:4873 , piece_id=LXL23F7EWSVJSTR2IXEMZXWFF2IUDA3LLZBEYPETRQWIOBILEG4A, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.518231417 (554h19m19.389314453s) [noise=true , address=145.79.227.47:4266 , piece_id=KOWY6SVD56MF5MCVPQN7M6I4TDIN4VEEMZYHVTUQVEDKSKDA5PZQ, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.520406589 (554h19m19.391489624s) [noise=true , address=145.79.227.9:4366 , piece_id=R4ZMQMHZ2RMW3PFE4XGBVL3QJDTJTTCEE5H3H3T3IRAVIPM4J2NQ, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.525375070 (554h19m19.396458105s) [noise=true , address=145.79.227.41:4085 , piece_id=GLBSELA3LZHQT7LV2JFVGAPGD5BBW67MENOUKNS22RFZ2EZ6JHPA, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.530370091 (554h19m19.401453136s) [noise=true , address=145.79.227.36:4518 , piece_id=XKGUW73TGFLUNCGBQDUJQYVDACDSCNWINAOTX4R42QN7VPZKMM3A, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.534822249 (554h19m19.405905294s) [noise=true , address=145.79.227.7:4295 , piece_id=SVKMYSBU4OS3ISBDTJSTD7H2VWYRRHEIU5FUNDNPPHYUZRY6TO2A, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.544482504 (554h19m19.415565549s) [noise=true , address=145.79.227.51:4410 , piece_id=YFUV7BGIVMN7AVAJO2CCLKO5O2AV42LHSEVO7DBYBBX4NWB2BSGA, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.580295799 (554h19m19.451378834s) [noise=true , address=145.79.227.14:4837 , piece_id=72K6MXJ5YGZWJUC6S7DQGIYDXDHYAKUXDCXJ7JRM5ARIOB5LCGOA, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.751560937 (554h19m19.622643972s) [noise=true , address=45.91.116.6:4975 , piece_id=VEEZX3RB3ZXHZI6UXZLFKCQ3XLIN7QJ7OMAHZSDP4IKKHFEG32NQ, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.757980792 (554h19m19.629063827s) [noise=true , address=45.91.116.9:4263 , piece_id=YELV2XVP4LMCOVRKWFRYND4NX237E4MUTB75LB6TDTK4FT62GJYA, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.981295543 (554h19m19.852378578s) [noise=true , address=45.91.116.12:4228 , piece_id=GQIOFAID4CORB7B5MBQ756KX25WQTK6GBPRBNAJXPDUZNS2NIKSQ, seg_pos={0 0}, upload=485986]: Done uploading piece. err: <nil> Dec 28 17:48:19.981318207 (554h19m19.852401242s) [seg_pos={0 0}, upload=485986]: Segment reached optimal threshold of 13 pieces. Dec 28 17:48:19.981528367 (554h19m19.852611422s) [noise=true , address=45.91.116.5:4297 , piece_id=IAKSE7B6THY3ZH3ZW4XBT6ASYZRVKVOR34EB4EL7O7VFKSEJWUGA, seg_pos={0 0}, upload=485986]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 17:48:19.981539327 (554h19m19.852622372s) [noise=true , address=45.91.116.5:4297 , piece_id=IAKSE7B6THY3ZH3ZW4XBT6ASYZRVKVOR34EB4EL7O7VFKSEJWUGA, seg_pos={0 0}, upload=485986]: Long tail cancellation due to optimal threshold Dec 28 17:48:19.982114764 (554h19m19.853197809s) [noise=true , address=45.91.116.8:4046 , piece_id=TDUBKJZ6FE4CCQLRNA42WEVU3HUBBVNE7RZIR2ROULFQ2OEOO2IQ, seg_pos={0 0}, upload=485986]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: read tcp 188.166.225.170:41064->45.91.116.8:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 17:48:19.982125123 (554h19m19.853208158s) [noise=true , address=45.91.116.8:4046 , piece_id=TDUBKJZ6FE4CCQLRNA42WEVU3HUBBVNE7RZIR2ROULFQ2OEOO2IQ, seg_pos={0 0}, upload=485986]: Long tail cancellation due to optimal threshold Dec 28 17:48:19.982379367 (554h19m19.853462412s) [noise=true , address=45.91.116.7:4010 , piece_id=BQMP4VMMN5D5HQCWPHAHCR5RU5VCYCUS6MPPHHO6DBCO5G5WHGKQ, seg_pos={0 0}, upload=485986]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:56668->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 17:48:19.982389176 (554h19m19.853472221s) [noise=true , address=45.91.116.7:4010 , piece_id=BQMP4VMMN5D5HQCWPHAHCR5RU5VCYCUS6MPPHHO6DBCO5G5WHGKQ, seg_pos={0 0}, upload=485986]: Long tail cancellation due to optimal threshold Dec 28 17:48:19.982414243 (554h19m19.853497278s) [seg_pos={0 0}, upload=485986]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:48:19.982501580 (554h19m19.853584635s) [upload=485986]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:48:19.982509785 (554h19m19.85359283s) [upload=485986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:48:20.281028137 (554h19m20.152111172s) [upload=485986]: Done uploading segments... Dec 28 17:48:20.281046102 (554h19m20.152129137s) [upload=485986]: Upload finished. err: <nil> Duration: 30.003502628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 25.331338622s trace: https://jaeger.dev.storj.io/trace/3d8b79377b19fbe5 FAIL (2025-12-28 17:48:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gin upload segment. Dec 28 17:44:30.976730747 (554h15m30.847813782s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.976740326 (554h15m30.847823361s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977238735 (554h15m30.84832177s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977395152 (554h15m30.848478187s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.979914119 (554h15m30.850997164s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.980878776 (554h15m30.851961811s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.981975264 (554h15m30.853058309s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982279864 (554h15m30.853362899s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982969287 (554h15m30.854052332s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.983667216 (554h15m30.854750271s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.984125969 (554h15m30.855209014s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985053625 (554h15m30.85613667s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985637347 (554h15m30.856720382s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985740974 (554h15m30.856824039s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.989521163 (554h15m30.860604198s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.990672796 (554h15m30.861755831s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.991767229 (554h15m30.862850284s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.993443933 (554h15m30.864526978s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:31.878310708 (554h15m31.749393753s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: ecclient: upload failed (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, address:193.149.164.143:4694): protocol: storage node overloaded, request limit: 2000 Dec 28 17:44:32.439383612 (554h15m32.310466647s) [upload=485932]: Next returned nil segment Dec 28 17:44:32.439397199 (554h15m32.310480234s) [upload=485932]: Waiting for error group managing segments... Dec 28 17:44:32.845179451 (554h15m32.716262486s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857045366 (554h15m32.728128401s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857359254 (554h15m32.728442289s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.907044634 (554h15m32.778127669s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.062870003 (554h15m32.933953048s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.152338938 (554h15m33.023421983s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.156658764 (554h15m33.027741799s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.196623976 (554h15m33.067707011s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.726903934 (554h15m33.597986979s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.746518434 (554h15m33.617601469s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.774407711 (554h15m33.645490756s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.823214066 (554h15m33.694297121s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.887815218 (554h15m33.758898253s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895889844 (554h15m33.766972879s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895905184 (554h15m33.766988229s) [seg_pos={0 0}, upload=485932]: Segment reached optimal threshold of 14 pieces. Dec 28 17:44:33.896119602 (554h15m33.767202637s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896129150 (554h15m33.767212205s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896347405 (554h15m33.76743045s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896385608 (554h15m33.767468653s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896453226 (554h15m33.767536261s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896465571 (554h15m33.767548626s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896487211 (554h15m33.767570246s) [seg_pos={0 0}, upload=485932]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:44:33.896575890 (554h15m33.767658936s) [upload=485932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:44:33.896582042 (554h15m33.767665087s) [upload=485932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:44:34.190733098 (554h15m34.061816143s) [upload=485932]: Done uploading segments... Dec 28 17:44:34.190758908 (554h15m34.061841943s) [upload=485932]: Upload finished. err: <nil> Duration: 25.331360484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X......................................................X....] duration: 44.417990324s trace: https://jaeger.dev.storj.io/trace/3d8b79377b19fbe5 FAIL (2025-12-28 17:48:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gin upload segment. Dec 28 17:44:30.976730747 (554h15m30.847813782s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.976740326 (554h15m30.847823361s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977238735 (554h15m30.84832177s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977395152 (554h15m30.848478187s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.979914119 (554h15m30.850997164s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.980878776 (554h15m30.851961811s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.981975264 (554h15m30.853058309s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982279864 (554h15m30.853362899s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982969287 (554h15m30.854052332s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.983667216 (554h15m30.854750271s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.984125969 (554h15m30.855209014s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985053625 (554h15m30.85613667s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985637347 (554h15m30.856720382s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985740974 (554h15m30.856824039s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.989521163 (554h15m30.860604198s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.990672796 (554h15m30.861755831s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.991767229 (554h15m30.862850284s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.993443933 (554h15m30.864526978s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:31.878310708 (554h15m31.749393753s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: ecclient: upload failed (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, address:193.149.164.143:4694): protocol: storage node overloaded, request limit: 2000 Dec 28 17:44:32.439383612 (554h15m32.310466647s) [upload=485932]: Next returned nil segment Dec 28 17:44:32.439397199 (554h15m32.310480234s) [upload=485932]: Waiting for error group managing segments... Dec 28 17:44:32.845179451 (554h15m32.716262486s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857045366 (554h15m32.728128401s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857359254 (554h15m32.728442289s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.907044634 (554h15m32.778127669s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.062870003 (554h15m32.933953048s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.152338938 (554h15m33.023421983s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.156658764 (554h15m33.027741799s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.196623976 (554h15m33.067707011s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.726903934 (554h15m33.597986979s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.746518434 (554h15m33.617601469s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.774407711 (554h15m33.645490756s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.823214066 (554h15m33.694297121s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.887815218 (554h15m33.758898253s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895889844 (554h15m33.766972879s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895905184 (554h15m33.766988229s) [seg_pos={0 0}, upload=485932]: Segment reached optimal threshold of 14 pieces. Dec 28 17:44:33.896119602 (554h15m33.767202637s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896129150 (554h15m33.767212205s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896347405 (554h15m33.76743045s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896385608 (554h15m33.767468653s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896453226 (554h15m33.767536261s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896465571 (554h15m33.767548626s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896487211 (554h15m33.767570246s) [seg_pos={0 0}, upload=485932]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:44:33.896575890 (554h15m33.767658936s) [upload=485932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:44:33.896582042 (554h15m33.767665087s) [upload=485932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:44:34.190733098 (554h15m34.061816143s) [upload=485932]: Done uploading segments... Dec 28 17:44:34.190758908 (554h15m34.061841943s) [upload=485932]: Upload finished. err: <nil> Duration: 44.418020602s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X....................X..X.............X...................] duration: 3.434517219s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:47:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.434550102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.........XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........X..] duration: 6.042413849s trace: https://jaeger.dev.storj.io/trace/18cd432242d4eef2 FAIL (2025-12-28 17:47:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 17:45:33.629871273 (554h16m33.500954338s) [upload=485946]: Starting upload Dec 28 17:45:33.629953219 (554h16m33.501036274s) [upload=485946]: Done starting upload Dec 28 17:45:33.630236999 (554h16m33.501320045s) [upload=485946]: Uploading segments... Dec 28 17:45:33.630286393 (554h16m33.501369428s) [upload=485946]: Got next segment. Inline: false Dec 28 17:45:33.630334946 (554h16m33.501417981s) [upload=485946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:45:33.630341328 (554h16m33.501424373s) [upload=485946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:45:33.874555285 (554h16m33.74563833s) [seg_pos={0 0}, upload=485946]: Begin upload segment... Dec 28 17:45:33.874599991 (554h16m33.745683036s) [seg_pos={0 0}, upload=485946]: Done begin upload segment. Dec 28 17:45:33.874633254 (554h16m33.745716289s) [noise=true , address=145.79.227.32:4836 , piece_id=BYCELTTNTNID3PVXGQYUHMSC4E5FUDQFUJID7JZ6HK5SMLQ6THDA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.874660115 (554h16m33.74574315s) [noise=true , address=145.79.227.2:4115 , piece_id=2ANMWNHYXB6MWXZHKFUBZY2KUIZMBBZCJN3G5RXIO4Q7XTUWCUZA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.875607119 (554h16m33.746690164s) [noise=true , address=77.90.31.16:4735 , piece_id=5TBC2ZRDO7Y5WGJ3M26LCGS53R6AVWUJI642Z542TOLFBIGDOL3Q, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.875832457 (554h16m33.746915503s) [noise=true , address=145.79.227.13:4802 , piece_id=WV3M343YUZDUHW6Q5M5YBXMDXP5RJVBAEHTSMCEJ4M6KTWMJLPZA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.876273638 (554h16m33.747356683s) [noise=true , address=145.79.227.47:4265 , piece_id=L4VFY7LEALCZQKN3SHUN7MMFZ3RX6WFE4X3DDSP7ONFPKCFGOU5Q, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.876695271 (554h16m33.747778316s) [noise=true , address=145.79.227.18:4007 , piece_id=7YDYGM5DNWBWAM7ZW4CZSEVL4K3QE25RMW2K7ARWAJWVU4SBH2SA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.877109169 (554h16m33.748192204s) [noise=true , address=145.79.227.45:4230 , piece_id=WK26KBYWD7XRCEWD4SBZVQW4KV4ZSIIQ7APQ2BYLDNZMZQW5F4XA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.878236004 (554h16m33.749319039s) [noise=true , address=45.91.116.8:4047 , piece_id=ZC3P475WMFK5KUL5X4SNEXSBPX6XHPJLIQW2OUV2FOSF5CTGA3DA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.878608834 (554h16m33.749691869s) [noise=true , address=145.79.227.34:4906 , piece_id=SDFKBXO5FHFTKDH6TCVLJ4SM5UZQGTHQRYN6CB7HCJ7MIHM4MT7A, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.879025909 (554h16m33.750108954s) [noise=true , address=145.79.227.11:4730 , piece_id=KGX52AKRNUGHB66C6KWOCUM3RHCU4Z2PTOILK3LVOBSY2YZBH2UA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.879491475 (554h16m33.75057452s) [noise=true , address=145.79.227.6:4258 , piece_id=A5XLF4JY7GQWZUFQCKO7RXUUDCFK3BOSDLAYMFQNSSSZ5TVMEA2Q, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.879966610 (554h16m33.751049645s) [noise=true , address=145.79.227.36:4519 , piece_id=L3CEJ4IPWWKPGMELYXMMXO7QSNSGXPJTMOPJ7X727MX2JW34DARQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.880318129 (554h16m33.751401164s) [noise=true , address=45.91.116.9:4262 , piece_id=7ZU7CPZOB6QWTY4ZHDUL5B5ZMKST2QFTTEA3KU5YDGAYRACFMMGA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.880630995 (554h16m33.75171403s) [noise=true , address=145.79.227.42:4123 , piece_id=BDBLJ6GIXBSD2G4GYVWDSNEE7EJVFZGYRRTRVE2A6OY644J5UVYQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.880971914 (554h16m33.752054949s) [noise=true , address=45.91.116.14:4191 , piece_id=JH6QZYLRGHFNOE3XGAOYPDFLZSE4I7P5W6AWM4YRIVM47RLR4NLQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.881275772 (554h16m33.752358817s) [noise=true , address=77.90.31.10:4520 , piece_id=PT3MQNRU6HXHAFGTB2OEDNH5WPPYL4N3SAPLNK5MPIT456E7GMDQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:35.578777493 (554h16m35.449860528s) [upload=485946]: Next returned nil segment Dec 28 17:45:35.578792942 (554h16m35.449875977s) [upload=485946]: Waiting for error group managing segments... Dec 28 17:45:35.885002320 (554h16m35.756085355s) [noise=true , address=145.79.227.45:4230 , piece_id=WK26KBYWD7XRCEWD4SBZVQW4KV4ZSIIQ7APQ2BYLDNZMZQW5F4XA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.886214658 (554h16m35.757297693s) [noise=true , address=145.79.227.32:4836 , piece_id=BYCELTTNTNID3PVXGQYUHMSC4E5FUDQFUJID7JZ6HK5SMLQ6THDA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.887916399 (554h16m35.758999444s) [noise=true , address=145.79.227.6:4258 , piece_id=A5XLF4JY7GQWZUFQCKO7RXUUDCFK3BOSDLAYMFQNSSSZ5TVMEA2Q, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.888994372 (554h16m35.760077407s) [noise=true , address=145.79.227.36:4519 , piece_id=L3CEJ4IPWWKPGMELYXMMXO7QSNSGXPJTMOPJ7X727MX2JW34DARQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.892394436 (554h16m35.763477481s) [noise=true , address=145.79.227.47:4265 , piece_id=L4VFY7LEALCZQKN3SHUN7MMFZ3RX6WFE4X3DDSP7ONFPKCFGOU5Q, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.899567064 (554h16m35.770650109s) [noise=true , address=145.79.227.11:4730 , piece_id=KGX52AKRNUGHB66C6KWOCUM3RHCU4Z2PTOILK3LVOBSY2YZBH2UA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.900216460 (554h16m35.771299505s) [noise=true , address=145.79.227.42:4123 , piece_id=BDBLJ6GIXBSD2G4GYVWDSNEE7EJVFZGYRRTRVE2A6OY644J5UVYQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.906146873 (554h16m35.777229908s) [noise=true , address=145.79.227.18:4007 , piece_id=7YDYGM5DNWBWAM7ZW4CZSEVL4K3QE25RMW2K7ARWAJWVU4SBH2SA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.906887633 (554h16m35.777970679s) [noise=true , address=145.79.227.2:4115 , piece_id=2ANMWNHYXB6MWXZHKFUBZY2KUIZMBBZCJN3G5RXIO4Q7XTUWCUZA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.915739058 (554h16m35.786822093s) [noise=true , address=145.79.227.13:4802 , piece_id=WV3M343YUZDUHW6Q5M5YBXMDXP5RJVBAEHTSMCEJ4M6KTWMJLPZA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.041472818 (554h16m35.912555863s) [noise=true , address=77.90.31.16:4735 , piece_id=5TBC2ZRDO7Y5WGJ3M26LCGS53R6AVWUJI642Z542TOLFBIGDOL3Q, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.151451066 (554h16m36.022534101s) [noise=true , address=45.91.116.14:4191 , piece_id=JH6QZYLRGHFNOE3XGAOYPDFLZSE4I7P5W6AWM4YRIVM47RLR4NLQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.232172725 (554h16m36.10325577s) [noise=true , address=45.91.116.9:4262 , piece_id=7ZU7CPZOB6QWTY4ZHDUL5B5ZMKST2QFTTEA3KU5YDGAYRACFMMGA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.232191862 (554h16m36.103274897s) [seg_pos={0 0}, upload=485946]: Segment reached optimal threshold of 13 pieces. Dec 28 17:45:36.232338230 (554h16m36.103421265s) [noise=true , address=145.79.227.34:4906 , piece_id=SDFKBXO5FHFTKDH6TCVLJ4SM5UZQGTHQRYN6CB7HCJ7MIHM4MT7A, seg_pos={0 0}, upload=485946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 17:45:36.232345554 (554h16m36.103428599s) [noise=true , address=145.79.227.34:4906 , piece_id=SDFKBXO5FHFTKDH6TCVLJ4SM5UZQGTHQRYN6CB7HCJ7MIHM4MT7A, seg_pos={0 0}, upload=485946]: Long tail cancellation due to optimal threshold Dec 28 17:45:36.232602363 (554h16m36.103685408s) [noise=true , address=45.91.116.8:4047 , piece_id=ZC3P475WMFK5KUL5X4SNEXSBPX6XHPJLIQW2OUV2FOSF5CTGA3DA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 17:45:36.232608936 (554h16m36.103691981s) [noise=true , address=45.91.116.8:4047 , piece_id=ZC3P475WMFK5KUL5X4SNEXSBPX6XHPJLIQW2OUV2FOSF5CTGA3DA, seg_pos={0 0}, upload=485946]: Long tail cancellation due to optimal threshold Dec 28 17:45:36.232704968 (554h16m36.103788003s) [noise=true , address=77.90.31.10:4520 , piece_id=PT3MQNRU6HXHAFGTB2OEDNH5WPPYL4N3SAPLNK5MPIT456E7GMDQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 28 17:45:36.232710108 (554h16m36.103793143s) [noise=true , address=77.90.31.10:4520 , piece_id=PT3MQNRU6HXHAFGTB2OEDNH5WPPYL4N3SAPLNK5MPIT456E7GMDQ, seg_pos={0 0}, upload=485946]: Long tail cancellation due to optimal threshold Dec 28 17:45:36.232727211 (554h16m36.103810246s) [seg_pos={0 0}, upload=485946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:45:36.232791884 (554h16m36.103874919s) [upload=485946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:45:36.232796212 (554h16m36.103879247s) [upload=485946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:45:36.521503654 (554h16m36.392586689s) [upload=485946]: Done uploading segments... Dec 28 17:45:36.521526828 (554h16m36.392609873s) [upload=485946]: Upload finished. err: <nil> Duration: 6.042443334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X......................................................X...] duration: 43.690745796s trace: https://jaeger.dev.storj.io/trace/3d8b79377b19fbe5 FAIL (2025-12-28 17:47:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gin upload segment. Dec 28 17:44:30.976730747 (554h15m30.847813782s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.976740326 (554h15m30.847823361s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977238735 (554h15m30.84832177s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977395152 (554h15m30.848478187s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.979914119 (554h15m30.850997164s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.980878776 (554h15m30.851961811s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.981975264 (554h15m30.853058309s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982279864 (554h15m30.853362899s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982969287 (554h15m30.854052332s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.983667216 (554h15m30.854750271s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.984125969 (554h15m30.855209014s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985053625 (554h15m30.85613667s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985637347 (554h15m30.856720382s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985740974 (554h15m30.856824039s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.989521163 (554h15m30.860604198s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.990672796 (554h15m30.861755831s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.991767229 (554h15m30.862850284s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.993443933 (554h15m30.864526978s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:31.878310708 (554h15m31.749393753s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: ecclient: upload failed (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, address:193.149.164.143:4694): protocol: storage node overloaded, request limit: 2000 Dec 28 17:44:32.439383612 (554h15m32.310466647s) [upload=485932]: Next returned nil segment Dec 28 17:44:32.439397199 (554h15m32.310480234s) [upload=485932]: Waiting for error group managing segments... Dec 28 17:44:32.845179451 (554h15m32.716262486s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857045366 (554h15m32.728128401s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857359254 (554h15m32.728442289s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.907044634 (554h15m32.778127669s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.062870003 (554h15m32.933953048s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.152338938 (554h15m33.023421983s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.156658764 (554h15m33.027741799s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.196623976 (554h15m33.067707011s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.726903934 (554h15m33.597986979s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.746518434 (554h15m33.617601469s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.774407711 (554h15m33.645490756s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.823214066 (554h15m33.694297121s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.887815218 (554h15m33.758898253s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895889844 (554h15m33.766972879s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895905184 (554h15m33.766988229s) [seg_pos={0 0}, upload=485932]: Segment reached optimal threshold of 14 pieces. Dec 28 17:44:33.896119602 (554h15m33.767202637s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896129150 (554h15m33.767212205s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896347405 (554h15m33.76743045s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896385608 (554h15m33.767468653s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896453226 (554h15m33.767536261s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896465571 (554h15m33.767548626s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896487211 (554h15m33.767570246s) [seg_pos={0 0}, upload=485932]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:44:33.896575890 (554h15m33.767658936s) [upload=485932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:44:33.896582042 (554h15m33.767665087s) [upload=485932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:44:34.190733098 (554h15m34.061816143s) [upload=485932]: Done uploading segments... Dec 28 17:44:34.190758908 (554h15m34.061841943s) [upload=485932]: Upload finished. err: <nil> Duration: 43.69076367s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X..X....................X..X.............X..................] duration: 3.537016375s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:47:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537047034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX.........XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........X.] duration: 6.795918604s trace: https://jaeger.dev.storj.io/trace/18cd432242d4eef2 FAIL (2025-12-28 17:46:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 17:45:33.629871273 (554h16m33.500954338s) [upload=485946]: Starting upload Dec 28 17:45:33.629953219 (554h16m33.501036274s) [upload=485946]: Done starting upload Dec 28 17:45:33.630236999 (554h16m33.501320045s) [upload=485946]: Uploading segments... Dec 28 17:45:33.630286393 (554h16m33.501369428s) [upload=485946]: Got next segment. Inline: false Dec 28 17:45:33.630334946 (554h16m33.501417981s) [upload=485946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:45:33.630341328 (554h16m33.501424373s) [upload=485946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:45:33.874555285 (554h16m33.74563833s) [seg_pos={0 0}, upload=485946]: Begin upload segment... Dec 28 17:45:33.874599991 (554h16m33.745683036s) [seg_pos={0 0}, upload=485946]: Done begin upload segment. Dec 28 17:45:33.874633254 (554h16m33.745716289s) [noise=true , address=145.79.227.32:4836 , piece_id=BYCELTTNTNID3PVXGQYUHMSC4E5FUDQFUJID7JZ6HK5SMLQ6THDA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.874660115 (554h16m33.74574315s) [noise=true , address=145.79.227.2:4115 , piece_id=2ANMWNHYXB6MWXZHKFUBZY2KUIZMBBZCJN3G5RXIO4Q7XTUWCUZA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.875607119 (554h16m33.746690164s) [noise=true , address=77.90.31.16:4735 , piece_id=5TBC2ZRDO7Y5WGJ3M26LCGS53R6AVWUJI642Z542TOLFBIGDOL3Q, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.875832457 (554h16m33.746915503s) [noise=true , address=145.79.227.13:4802 , piece_id=WV3M343YUZDUHW6Q5M5YBXMDXP5RJVBAEHTSMCEJ4M6KTWMJLPZA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.876273638 (554h16m33.747356683s) [noise=true , address=145.79.227.47:4265 , piece_id=L4VFY7LEALCZQKN3SHUN7MMFZ3RX6WFE4X3DDSP7ONFPKCFGOU5Q, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.876695271 (554h16m33.747778316s) [noise=true , address=145.79.227.18:4007 , piece_id=7YDYGM5DNWBWAM7ZW4CZSEVL4K3QE25RMW2K7ARWAJWVU4SBH2SA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.877109169 (554h16m33.748192204s) [noise=true , address=145.79.227.45:4230 , piece_id=WK26KBYWD7XRCEWD4SBZVQW4KV4ZSIIQ7APQ2BYLDNZMZQW5F4XA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.878236004 (554h16m33.749319039s) [noise=true , address=45.91.116.8:4047 , piece_id=ZC3P475WMFK5KUL5X4SNEXSBPX6XHPJLIQW2OUV2FOSF5CTGA3DA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.878608834 (554h16m33.749691869s) [noise=true , address=145.79.227.34:4906 , piece_id=SDFKBXO5FHFTKDH6TCVLJ4SM5UZQGTHQRYN6CB7HCJ7MIHM4MT7A, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.879025909 (554h16m33.750108954s) [noise=true , address=145.79.227.11:4730 , piece_id=KGX52AKRNUGHB66C6KWOCUM3RHCU4Z2PTOILK3LVOBSY2YZBH2UA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.879491475 (554h16m33.75057452s) [noise=true , address=145.79.227.6:4258 , piece_id=A5XLF4JY7GQWZUFQCKO7RXUUDCFK3BOSDLAYMFQNSSSZ5TVMEA2Q, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.879966610 (554h16m33.751049645s) [noise=true , address=145.79.227.36:4519 , piece_id=L3CEJ4IPWWKPGMELYXMMXO7QSNSGXPJTMOPJ7X727MX2JW34DARQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.880318129 (554h16m33.751401164s) [noise=true , address=45.91.116.9:4262 , piece_id=7ZU7CPZOB6QWTY4ZHDUL5B5ZMKST2QFTTEA3KU5YDGAYRACFMMGA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.880630995 (554h16m33.75171403s) [noise=true , address=145.79.227.42:4123 , piece_id=BDBLJ6GIXBSD2G4GYVWDSNEE7EJVFZGYRRTRVE2A6OY644J5UVYQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.880971914 (554h16m33.752054949s) [noise=true , address=45.91.116.14:4191 , piece_id=JH6QZYLRGHFNOE3XGAOYPDFLZSE4I7P5W6AWM4YRIVM47RLR4NLQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.881275772 (554h16m33.752358817s) [noise=true , address=77.90.31.10:4520 , piece_id=PT3MQNRU6HXHAFGTB2OEDNH5WPPYL4N3SAPLNK5MPIT456E7GMDQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:35.578777493 (554h16m35.449860528s) [upload=485946]: Next returned nil segment Dec 28 17:45:35.578792942 (554h16m35.449875977s) [upload=485946]: Waiting for error group managing segments... Dec 28 17:45:35.885002320 (554h16m35.756085355s) [noise=true , address=145.79.227.45:4230 , piece_id=WK26KBYWD7XRCEWD4SBZVQW4KV4ZSIIQ7APQ2BYLDNZMZQW5F4XA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.886214658 (554h16m35.757297693s) [noise=true , address=145.79.227.32:4836 , piece_id=BYCELTTNTNID3PVXGQYUHMSC4E5FUDQFUJID7JZ6HK5SMLQ6THDA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.887916399 (554h16m35.758999444s) [noise=true , address=145.79.227.6:4258 , piece_id=A5XLF4JY7GQWZUFQCKO7RXUUDCFK3BOSDLAYMFQNSSSZ5TVMEA2Q, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.888994372 (554h16m35.760077407s) [noise=true , address=145.79.227.36:4519 , piece_id=L3CEJ4IPWWKPGMELYXMMXO7QSNSGXPJTMOPJ7X727MX2JW34DARQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.892394436 (554h16m35.763477481s) [noise=true , address=145.79.227.47:4265 , piece_id=L4VFY7LEALCZQKN3SHUN7MMFZ3RX6WFE4X3DDSP7ONFPKCFGOU5Q, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.899567064 (554h16m35.770650109s) [noise=true , address=145.79.227.11:4730 , piece_id=KGX52AKRNUGHB66C6KWOCUM3RHCU4Z2PTOILK3LVOBSY2YZBH2UA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.900216460 (554h16m35.771299505s) [noise=true , address=145.79.227.42:4123 , piece_id=BDBLJ6GIXBSD2G4GYVWDSNEE7EJVFZGYRRTRVE2A6OY644J5UVYQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.906146873 (554h16m35.777229908s) [noise=true , address=145.79.227.18:4007 , piece_id=7YDYGM5DNWBWAM7ZW4CZSEVL4K3QE25RMW2K7ARWAJWVU4SBH2SA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.906887633 (554h16m35.777970679s) [noise=true , address=145.79.227.2:4115 , piece_id=2ANMWNHYXB6MWXZHKFUBZY2KUIZMBBZCJN3G5RXIO4Q7XTUWCUZA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.915739058 (554h16m35.786822093s) [noise=true , address=145.79.227.13:4802 , piece_id=WV3M343YUZDUHW6Q5M5YBXMDXP5RJVBAEHTSMCEJ4M6KTWMJLPZA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.041472818 (554h16m35.912555863s) [noise=true , address=77.90.31.16:4735 , piece_id=5TBC2ZRDO7Y5WGJ3M26LCGS53R6AVWUJI642Z542TOLFBIGDOL3Q, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.151451066 (554h16m36.022534101s) [noise=true , address=45.91.116.14:4191 , piece_id=JH6QZYLRGHFNOE3XGAOYPDFLZSE4I7P5W6AWM4YRIVM47RLR4NLQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.232172725 (554h16m36.10325577s) [noise=true , address=45.91.116.9:4262 , piece_id=7ZU7CPZOB6QWTY4ZHDUL5B5ZMKST2QFTTEA3KU5YDGAYRACFMMGA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.232191862 (554h16m36.103274897s) [seg_pos={0 0}, upload=485946]: Segment reached optimal threshold of 13 pieces. Dec 28 17:45:36.232338230 (554h16m36.103421265s) [noise=true , address=145.79.227.34:4906 , piece_id=SDFKBXO5FHFTKDH6TCVLJ4SM5UZQGTHQRYN6CB7HCJ7MIHM4MT7A, seg_pos={0 0}, upload=485946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 17:45:36.232345554 (554h16m36.103428599s) [noise=true , address=145.79.227.34:4906 , piece_id=SDFKBXO5FHFTKDH6TCVLJ4SM5UZQGTHQRYN6CB7HCJ7MIHM4MT7A, seg_pos={0 0}, upload=485946]: Long tail cancellation due to optimal threshold Dec 28 17:45:36.232602363 (554h16m36.103685408s) [noise=true , address=45.91.116.8:4047 , piece_id=ZC3P475WMFK5KUL5X4SNEXSBPX6XHPJLIQW2OUV2FOSF5CTGA3DA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 17:45:36.232608936 (554h16m36.103691981s) [noise=true , address=45.91.116.8:4047 , piece_id=ZC3P475WMFK5KUL5X4SNEXSBPX6XHPJLIQW2OUV2FOSF5CTGA3DA, seg_pos={0 0}, upload=485946]: Long tail cancellation due to optimal threshold Dec 28 17:45:36.232704968 (554h16m36.103788003s) [noise=true , address=77.90.31.10:4520 , piece_id=PT3MQNRU6HXHAFGTB2OEDNH5WPPYL4N3SAPLNK5MPIT456E7GMDQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 28 17:45:36.232710108 (554h16m36.103793143s) [noise=true , address=77.90.31.10:4520 , piece_id=PT3MQNRU6HXHAFGTB2OEDNH5WPPYL4N3SAPLNK5MPIT456E7GMDQ, seg_pos={0 0}, upload=485946]: Long tail cancellation due to optimal threshold Dec 28 17:45:36.232727211 (554h16m36.103810246s) [seg_pos={0 0}, upload=485946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:45:36.232791884 (554h16m36.103874919s) [upload=485946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:45:36.232796212 (554h16m36.103879247s) [upload=485946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:45:36.521503654 (554h16m36.392586689s) [upload=485946]: Done uploading segments... Dec 28 17:45:36.521526828 (554h16m36.392609873s) [upload=485946]: Upload finished. err: <nil> Duration: 6.795946758s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X......................................................X..] duration: 33.063663626s trace: https://jaeger.dev.storj.io/trace/3d8b79377b19fbe5 FAIL (2025-12-28 17:46:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gin upload segment. Dec 28 17:44:30.976730747 (554h15m30.847813782s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.976740326 (554h15m30.847823361s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977238735 (554h15m30.84832177s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977395152 (554h15m30.848478187s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.979914119 (554h15m30.850997164s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.980878776 (554h15m30.851961811s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.981975264 (554h15m30.853058309s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982279864 (554h15m30.853362899s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982969287 (554h15m30.854052332s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.983667216 (554h15m30.854750271s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.984125969 (554h15m30.855209014s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985053625 (554h15m30.85613667s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985637347 (554h15m30.856720382s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985740974 (554h15m30.856824039s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.989521163 (554h15m30.860604198s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.990672796 (554h15m30.861755831s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.991767229 (554h15m30.862850284s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.993443933 (554h15m30.864526978s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:31.878310708 (554h15m31.749393753s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: ecclient: upload failed (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, address:193.149.164.143:4694): protocol: storage node overloaded, request limit: 2000 Dec 28 17:44:32.439383612 (554h15m32.310466647s) [upload=485932]: Next returned nil segment Dec 28 17:44:32.439397199 (554h15m32.310480234s) [upload=485932]: Waiting for error group managing segments... Dec 28 17:44:32.845179451 (554h15m32.716262486s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857045366 (554h15m32.728128401s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857359254 (554h15m32.728442289s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.907044634 (554h15m32.778127669s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.062870003 (554h15m32.933953048s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.152338938 (554h15m33.023421983s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.156658764 (554h15m33.027741799s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.196623976 (554h15m33.067707011s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.726903934 (554h15m33.597986979s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.746518434 (554h15m33.617601469s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.774407711 (554h15m33.645490756s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.823214066 (554h15m33.694297121s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.887815218 (554h15m33.758898253s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895889844 (554h15m33.766972879s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895905184 (554h15m33.766988229s) [seg_pos={0 0}, upload=485932]: Segment reached optimal threshold of 14 pieces. Dec 28 17:44:33.896119602 (554h15m33.767202637s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896129150 (554h15m33.767212205s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896347405 (554h15m33.76743045s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896385608 (554h15m33.767468653s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896453226 (554h15m33.767536261s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896465571 (554h15m33.767548626s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896487211 (554h15m33.767570246s) [seg_pos={0 0}, upload=485932]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:44:33.896575890 (554h15m33.767658936s) [upload=485932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:44:33.896582042 (554h15m33.767665087s) [upload=485932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:44:34.190733098 (554h15m34.061816143s) [upload=485932]: Done uploading segments... Dec 28 17:44:34.190758908 (554h15m34.061841943s) [upload=485932]: Upload finished. err: <nil> Duration: 33.063682893s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X..X....................X..X.............X.................] duration: 3.534426463s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:46:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534468243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX.........XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........X] duration: 30.003549755s trace: https://jaeger.dev.storj.io/trace/18cd432242d4eef2 FAIL (2025-12-28 17:46:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 17:45:33.629871273 (554h16m33.500954338s) [upload=485946]: Starting upload Dec 28 17:45:33.629953219 (554h16m33.501036274s) [upload=485946]: Done starting upload Dec 28 17:45:33.630236999 (554h16m33.501320045s) [upload=485946]: Uploading segments... Dec 28 17:45:33.630286393 (554h16m33.501369428s) [upload=485946]: Got next segment. Inline: false Dec 28 17:45:33.630334946 (554h16m33.501417981s) [upload=485946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:45:33.630341328 (554h16m33.501424373s) [upload=485946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:45:33.874555285 (554h16m33.74563833s) [seg_pos={0 0}, upload=485946]: Begin upload segment... Dec 28 17:45:33.874599991 (554h16m33.745683036s) [seg_pos={0 0}, upload=485946]: Done begin upload segment. Dec 28 17:45:33.874633254 (554h16m33.745716289s) [noise=true , address=145.79.227.32:4836 , piece_id=BYCELTTNTNID3PVXGQYUHMSC4E5FUDQFUJID7JZ6HK5SMLQ6THDA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.874660115 (554h16m33.74574315s) [noise=true , address=145.79.227.2:4115 , piece_id=2ANMWNHYXB6MWXZHKFUBZY2KUIZMBBZCJN3G5RXIO4Q7XTUWCUZA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.875607119 (554h16m33.746690164s) [noise=true , address=77.90.31.16:4735 , piece_id=5TBC2ZRDO7Y5WGJ3M26LCGS53R6AVWUJI642Z542TOLFBIGDOL3Q, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.875832457 (554h16m33.746915503s) [noise=true , address=145.79.227.13:4802 , piece_id=WV3M343YUZDUHW6Q5M5YBXMDXP5RJVBAEHTSMCEJ4M6KTWMJLPZA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.876273638 (554h16m33.747356683s) [noise=true , address=145.79.227.47:4265 , piece_id=L4VFY7LEALCZQKN3SHUN7MMFZ3RX6WFE4X3DDSP7ONFPKCFGOU5Q, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.876695271 (554h16m33.747778316s) [noise=true , address=145.79.227.18:4007 , piece_id=7YDYGM5DNWBWAM7ZW4CZSEVL4K3QE25RMW2K7ARWAJWVU4SBH2SA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.877109169 (554h16m33.748192204s) [noise=true , address=145.79.227.45:4230 , piece_id=WK26KBYWD7XRCEWD4SBZVQW4KV4ZSIIQ7APQ2BYLDNZMZQW5F4XA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.878236004 (554h16m33.749319039s) [noise=true , address=45.91.116.8:4047 , piece_id=ZC3P475WMFK5KUL5X4SNEXSBPX6XHPJLIQW2OUV2FOSF5CTGA3DA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.878608834 (554h16m33.749691869s) [noise=true , address=145.79.227.34:4906 , piece_id=SDFKBXO5FHFTKDH6TCVLJ4SM5UZQGTHQRYN6CB7HCJ7MIHM4MT7A, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.879025909 (554h16m33.750108954s) [noise=true , address=145.79.227.11:4730 , piece_id=KGX52AKRNUGHB66C6KWOCUM3RHCU4Z2PTOILK3LVOBSY2YZBH2UA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.879491475 (554h16m33.75057452s) [noise=true , address=145.79.227.6:4258 , piece_id=A5XLF4JY7GQWZUFQCKO7RXUUDCFK3BOSDLAYMFQNSSSZ5TVMEA2Q, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.879966610 (554h16m33.751049645s) [noise=true , address=145.79.227.36:4519 , piece_id=L3CEJ4IPWWKPGMELYXMMXO7QSNSGXPJTMOPJ7X727MX2JW34DARQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.880318129 (554h16m33.751401164s) [noise=true , address=45.91.116.9:4262 , piece_id=7ZU7CPZOB6QWTY4ZHDUL5B5ZMKST2QFTTEA3KU5YDGAYRACFMMGA, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.880630995 (554h16m33.75171403s) [noise=true , address=145.79.227.42:4123 , piece_id=BDBLJ6GIXBSD2G4GYVWDSNEE7EJVFZGYRRTRVE2A6OY644J5UVYQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.880971914 (554h16m33.752054949s) [noise=true , address=45.91.116.14:4191 , piece_id=JH6QZYLRGHFNOE3XGAOYPDFLZSE4I7P5W6AWM4YRIVM47RLR4NLQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:33.881275772 (554h16m33.752358817s) [noise=true , address=77.90.31.10:4520 , piece_id=PT3MQNRU6HXHAFGTB2OEDNH5WPPYL4N3SAPLNK5MPIT456E7GMDQ, seg_pos={0 0}, upload=485946]: Uploading piece... Dec 28 17:45:35.578777493 (554h16m35.449860528s) [upload=485946]: Next returned nil segment Dec 28 17:45:35.578792942 (554h16m35.449875977s) [upload=485946]: Waiting for error group managing segments... Dec 28 17:45:35.885002320 (554h16m35.756085355s) [noise=true , address=145.79.227.45:4230 , piece_id=WK26KBYWD7XRCEWD4SBZVQW4KV4ZSIIQ7APQ2BYLDNZMZQW5F4XA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.886214658 (554h16m35.757297693s) [noise=true , address=145.79.227.32:4836 , piece_id=BYCELTTNTNID3PVXGQYUHMSC4E5FUDQFUJID7JZ6HK5SMLQ6THDA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.887916399 (554h16m35.758999444s) [noise=true , address=145.79.227.6:4258 , piece_id=A5XLF4JY7GQWZUFQCKO7RXUUDCFK3BOSDLAYMFQNSSSZ5TVMEA2Q, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.888994372 (554h16m35.760077407s) [noise=true , address=145.79.227.36:4519 , piece_id=L3CEJ4IPWWKPGMELYXMMXO7QSNSGXPJTMOPJ7X727MX2JW34DARQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.892394436 (554h16m35.763477481s) [noise=true , address=145.79.227.47:4265 , piece_id=L4VFY7LEALCZQKN3SHUN7MMFZ3RX6WFE4X3DDSP7ONFPKCFGOU5Q, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.899567064 (554h16m35.770650109s) [noise=true , address=145.79.227.11:4730 , piece_id=KGX52AKRNUGHB66C6KWOCUM3RHCU4Z2PTOILK3LVOBSY2YZBH2UA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.900216460 (554h16m35.771299505s) [noise=true , address=145.79.227.42:4123 , piece_id=BDBLJ6GIXBSD2G4GYVWDSNEE7EJVFZGYRRTRVE2A6OY644J5UVYQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.906146873 (554h16m35.777229908s) [noise=true , address=145.79.227.18:4007 , piece_id=7YDYGM5DNWBWAM7ZW4CZSEVL4K3QE25RMW2K7ARWAJWVU4SBH2SA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.906887633 (554h16m35.777970679s) [noise=true , address=145.79.227.2:4115 , piece_id=2ANMWNHYXB6MWXZHKFUBZY2KUIZMBBZCJN3G5RXIO4Q7XTUWCUZA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:35.915739058 (554h16m35.786822093s) [noise=true , address=145.79.227.13:4802 , piece_id=WV3M343YUZDUHW6Q5M5YBXMDXP5RJVBAEHTSMCEJ4M6KTWMJLPZA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.041472818 (554h16m35.912555863s) [noise=true , address=77.90.31.16:4735 , piece_id=5TBC2ZRDO7Y5WGJ3M26LCGS53R6AVWUJI642Z542TOLFBIGDOL3Q, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.151451066 (554h16m36.022534101s) [noise=true , address=45.91.116.14:4191 , piece_id=JH6QZYLRGHFNOE3XGAOYPDFLZSE4I7P5W6AWM4YRIVM47RLR4NLQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.232172725 (554h16m36.10325577s) [noise=true , address=45.91.116.9:4262 , piece_id=7ZU7CPZOB6QWTY4ZHDUL5B5ZMKST2QFTTEA3KU5YDGAYRACFMMGA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: <nil> Dec 28 17:45:36.232191862 (554h16m36.103274897s) [seg_pos={0 0}, upload=485946]: Segment reached optimal threshold of 13 pieces. Dec 28 17:45:36.232338230 (554h16m36.103421265s) [noise=true , address=145.79.227.34:4906 , piece_id=SDFKBXO5FHFTKDH6TCVLJ4SM5UZQGTHQRYN6CB7HCJ7MIHM4MT7A, seg_pos={0 0}, upload=485946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 17:45:36.232345554 (554h16m36.103428599s) [noise=true , address=145.79.227.34:4906 , piece_id=SDFKBXO5FHFTKDH6TCVLJ4SM5UZQGTHQRYN6CB7HCJ7MIHM4MT7A, seg_pos={0 0}, upload=485946]: Long tail cancellation due to optimal threshold Dec 28 17:45:36.232602363 (554h16m36.103685408s) [noise=true , address=45.91.116.8:4047 , piece_id=ZC3P475WMFK5KUL5X4SNEXSBPX6XHPJLIQW2OUV2FOSF5CTGA3DA, seg_pos={0 0}, upload=485946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 17:45:36.232608936 (554h16m36.103691981s) [noise=true , address=45.91.116.8:4047 , piece_id=ZC3P475WMFK5KUL5X4SNEXSBPX6XHPJLIQW2OUV2FOSF5CTGA3DA, seg_pos={0 0}, upload=485946]: Long tail cancellation due to optimal threshold Dec 28 17:45:36.232704968 (554h16m36.103788003s) [noise=true , address=77.90.31.10:4520 , piece_id=PT3MQNRU6HXHAFGTB2OEDNH5WPPYL4N3SAPLNK5MPIT456E7GMDQ, seg_pos={0 0}, upload=485946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 28 17:45:36.232710108 (554h16m36.103793143s) [noise=true , address=77.90.31.10:4520 , piece_id=PT3MQNRU6HXHAFGTB2OEDNH5WPPYL4N3SAPLNK5MPIT456E7GMDQ, seg_pos={0 0}, upload=485946]: Long tail cancellation due to optimal threshold Dec 28 17:45:36.232727211 (554h16m36.103810246s) [seg_pos={0 0}, upload=485946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:45:36.232791884 (554h16m36.103874919s) [upload=485946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:45:36.232796212 (554h16m36.103879247s) [upload=485946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:45:36.521503654 (554h16m36.392586689s) [upload=485946]: Done uploading segments... Dec 28 17:45:36.521526828 (554h16m36.392609873s) [upload=485946]: Upload finished. err: <nil> Duration: 30.003572188s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X..X....................X..X.............X................] duration: 3.440957326s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:45:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440998043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X......................................................X.] duration: 15.467164747s trace: https://jaeger.dev.storj.io/trace/3d8b79377b19fbe5 FAIL (2025-12-28 17:45:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gin upload segment. Dec 28 17:44:30.976730747 (554h15m30.847813782s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.976740326 (554h15m30.847823361s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977238735 (554h15m30.84832177s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977395152 (554h15m30.848478187s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.979914119 (554h15m30.850997164s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.980878776 (554h15m30.851961811s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.981975264 (554h15m30.853058309s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982279864 (554h15m30.853362899s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982969287 (554h15m30.854052332s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.983667216 (554h15m30.854750271s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.984125969 (554h15m30.855209014s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985053625 (554h15m30.85613667s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985637347 (554h15m30.856720382s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985740974 (554h15m30.856824039s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.989521163 (554h15m30.860604198s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.990672796 (554h15m30.861755831s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.991767229 (554h15m30.862850284s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.993443933 (554h15m30.864526978s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:31.878310708 (554h15m31.749393753s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: ecclient: upload failed (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, address:193.149.164.143:4694): protocol: storage node overloaded, request limit: 2000 Dec 28 17:44:32.439383612 (554h15m32.310466647s) [upload=485932]: Next returned nil segment Dec 28 17:44:32.439397199 (554h15m32.310480234s) [upload=485932]: Waiting for error group managing segments... Dec 28 17:44:32.845179451 (554h15m32.716262486s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857045366 (554h15m32.728128401s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857359254 (554h15m32.728442289s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.907044634 (554h15m32.778127669s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.062870003 (554h15m32.933953048s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.152338938 (554h15m33.023421983s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.156658764 (554h15m33.027741799s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.196623976 (554h15m33.067707011s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.726903934 (554h15m33.597986979s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.746518434 (554h15m33.617601469s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.774407711 (554h15m33.645490756s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.823214066 (554h15m33.694297121s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.887815218 (554h15m33.758898253s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895889844 (554h15m33.766972879s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895905184 (554h15m33.766988229s) [seg_pos={0 0}, upload=485932]: Segment reached optimal threshold of 14 pieces. Dec 28 17:44:33.896119602 (554h15m33.767202637s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896129150 (554h15m33.767212205s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896347405 (554h15m33.76743045s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896385608 (554h15m33.767468653s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896453226 (554h15m33.767536261s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896465571 (554h15m33.767548626s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896487211 (554h15m33.767570246s) [seg_pos={0 0}, upload=485932]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:44:33.896575890 (554h15m33.767658936s) [upload=485932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:44:33.896582042 (554h15m33.767665087s) [upload=485932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:44:34.190733098 (554h15m34.061816143s) [upload=485932]: Done uploading segments... Dec 28 17:44:34.190758908 (554h15m34.061841943s) [upload=485932]: Upload finished. err: <nil> Duration: 15.467179566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X......................................................X] duration: 1m0.00288903s trace: https://jaeger.dev.storj.io/trace/3d8b79377b19fbe5 FAIL (2025-12-28 17:45:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gin upload segment. Dec 28 17:44:30.976730747 (554h15m30.847813782s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.976740326 (554h15m30.847823361s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977238735 (554h15m30.84832177s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.977395152 (554h15m30.848478187s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.979914119 (554h15m30.850997164s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.980878776 (554h15m30.851961811s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.981975264 (554h15m30.853058309s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982279864 (554h15m30.853362899s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.982969287 (554h15m30.854052332s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.983667216 (554h15m30.854750271s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.984125969 (554h15m30.855209014s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985053625 (554h15m30.85613667s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985637347 (554h15m30.856720382s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.985740974 (554h15m30.856824039s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.989521163 (554h15m30.860604198s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.990672796 (554h15m30.861755831s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.991767229 (554h15m30.862850284s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:30.993443933 (554h15m30.864526978s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Uploading piece... Dec 28 17:44:31.878310708 (554h15m31.749393753s) [noise=true , address=193.149.164.143:4694 , piece_id=CTCDGDQPQEKMGNZ675JWD2QNPNKAPX2RJWKIIXHWADX2ZTQ5NAVA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: ecclient: upload failed (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, address:193.149.164.143:4694): protocol: storage node overloaded, request limit: 2000 Dec 28 17:44:32.439383612 (554h15m32.310466647s) [upload=485932]: Next returned nil segment Dec 28 17:44:32.439397199 (554h15m32.310480234s) [upload=485932]: Waiting for error group managing segments... Dec 28 17:44:32.845179451 (554h15m32.716262486s) [noise=true , address=145.79.227.8:4361 , piece_id=WULWD7IL723CCKBGZ2TMCHXNDTOBRFBX5MTFGYRT4L2OIDD2CPIQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857045366 (554h15m32.728128401s) [noise=true , address=145.79.227.36:4541 , piece_id=3LLZ7UNXPTXTCUXJ2VW5NNRHTW6NZKGX3YDEO2HFBUBVX64PWMTQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.857359254 (554h15m32.728442289s) [noise=true , address=145.79.227.9:4382 , piece_id=C57ZDV5XXYAO7XAVMQ4WXSWKO5L6ZY2X6HVF4OJ2Q3R77RSW4VZQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:32.907044634 (554h15m32.778127669s) [noise=true , address=145.79.227.26:4683 , piece_id=WYSDW2JJJVJANPWZH6WGWEMFCGVPKJDSUQPEMHIMDDVIZGZAW52Q, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.062870003 (554h15m32.933953048s) [noise=true , address=77.90.31.21:4881 , piece_id=FVMUG6I4G4S6BDOI6ZHGQR4MDHIBBA6C234QTP2TGN4XFNZWDBSQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.152338938 (554h15m33.023421983s) [noise=true , address=77.90.31.11:4561 , piece_id=T2HFD2FP7CVTXCQX6NRM47RNZOSAXTDOMHK6B34GBPERPXYICSYA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.156658764 (554h15m33.027741799s) [noise=true , address=77.90.31.14:4688 , piece_id=7FJ3VZPSHYBLGNO4GIE33YIHFMPEO3AGRZWW5OP7FNKTBMFZG4AA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.196623976 (554h15m33.067707011s) [noise=true , address=45.91.116.5:4312 , piece_id=3PKG4EYUZHRRWVMYF76DNLEYYCEY55YKWG6IX5ER4RN4ILKUY2JQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.726903934 (554h15m33.597986979s) [noise=true , address=145.79.227.44:4205 , piece_id=H4LNEKPJNIEY2WCZQHEWEGL776EEHYHHXVITYEIHOSJ53M6YUIXA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.746518434 (554h15m33.617601469s) [noise=true , address=145.79.227.44:4213 , piece_id=3AHN2S4I2KO35ZIE55PJHANIWOZGUMTJ4LXXJNTFSUT3R75BO3WA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.774407711 (554h15m33.645490756s) [noise=true , address=77.90.31.13:4637 , piece_id=3UAUMN6YLBL6673RFSBLH36OPTN6FDSUJ72K6SKKVVS4BOI5LW3A, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.823214066 (554h15m33.694297121s) [noise=true , address=145.79.227.43:4167 , piece_id=JU7HWRDCXFPA5ZGPVBY6EYE26GGPQW5MBRLSWVPCNQZVZQEQZXWA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.887815218 (554h15m33.758898253s) [noise=true , address=145.79.227.29:4959 , piece_id=2S4KSANT7NS2APATTWL33G6DUHQ4CRGIDDH356N3HJWU45RNC7CA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895889844 (554h15m33.766972879s) [noise=true , address=77.90.31.19:4857 , piece_id=3UXB767N3H5DFHWZK67FNYKKDH5UAGFZOL26D5DWNVI76BXEDGNA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: <nil> Dec 28 17:44:33.895905184 (554h15m33.766988229s) [seg_pos={0 0}, upload=485932]: Segment reached optimal threshold of 14 pieces. Dec 28 17:44:33.896119602 (554h15m33.767202637s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896129150 (554h15m33.767212205s) [noise=true , address=145.79.227.34:4929 , piece_id=7QNUVFZFMHDI3W35RJK6JFA7XIKNFIOSML4OTXVAN6VZ6DED5YCA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896347405 (554h15m33.76743045s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896385608 (554h15m33.767468653s) [noise=true , address=145.79.227.33:4902 , piece_id=LUWTWW43WDKROO2QHPWGZMOV6FQECEXK7FRHN6LZODPBZCBM7IDQ, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896453226 (554h15m33.767536261s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 28 17:44:33.896465571 (554h15m33.767548626s) [noise=true , address=77.90.31.15:4701 , piece_id=OJTV5DGERKXNBUSPKXBFXOCDFMZZUKWL5NVBFVXEK6MWXX2MVJFA, seg_pos={0 0}, upload=485932]: Long tail cancellation due to optimal threshold Dec 28 17:44:33.896487211 (554h15m33.767570246s) [seg_pos={0 0}, upload=485932]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 17:44:33.896575890 (554h15m33.767658936s) [upload=485932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:44:33.896582042 (554h15m33.767665087s) [upload=485932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:44:34.190733098 (554h15m34.061816143s) [upload=485932]: Done uploading segments... Dec 28 17:44:34.190758908 (554h15m34.061841943s) [upload=485932]: Upload finished. err: <nil> Duration: 1m0.0029157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X..X....................X..X.............X...............] duration: 3.406825534s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:44:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.406852464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX.........XXX...XX...X.X..XXX.XXX...XX...XX.X..XX........] duration: 6.224212636s trace: https://jaeger.dev.storj.io/trace/3a541c3d76227692 FAIL (2025-12-28 17:44:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A) context deadline exceeded ======LOGS====== Dec 28 17:39:36.354400600 (554h10m36.225483635s) [upload=485859]: Starting upload Dec 28 17:39:36.354469531 (554h10m36.225552576s) [upload=485859]: Done starting upload Dec 28 17:39:36.354642701 (554h10m36.225725746s) [upload=485859]: Uploading segments... Dec 28 17:39:36.354914058 (554h10m36.225997093s) [upload=485859]: Got next segment. Inline: false Dec 28 17:39:36.354956999 (554h10m36.226040034s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:39:36.354961368 (554h10m36.226044413s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:39:36.601247130 (554h10m36.472330185s) [seg_pos={0 0}, upload=485859]: Begin upload segment... Dec 28 17:39:36.601335769 (554h10m36.472418804s) [seg_pos={0 0}, upload=485859]: Done begin upload segment. Dec 28 17:39:36.601377288 (554h10m36.472460323s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.601413357 (554h10m36.472496402s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602179746 (554h10m36.473262801s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602519904 (554h10m36.473602949s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603138352 (554h10m36.474221387s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603634567 (554h10m36.474717612s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603905994 (554h10m36.474989049s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604115352 (554h10m36.475198387s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604430973 (554h10m36.475514018s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604675990 (554h10m36.475759025s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604768907 (554h10m36.475851952s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.605244272 (554h10m36.476327317s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.611659136 (554h10m36.482742181s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.613275875 (554h10m36.484358921s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616040870 (554h10m36.487123915s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616651533 (554h10m36.487734568s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:38.318685603 (554h10m38.189768648s) [upload=485859]: Next returned nil segment Dec 28 17:39:38.318702345 (554h10m38.18978538s) [upload=485859]: Waiting for error group managing segments... Dec 28 17:39:38.601154966 (554h10m38.472238001s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.605166786 (554h10m38.476249831s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.608587720 (554h10m38.479670755s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.613615423 (554h10m38.484698468s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.618905937 (554h10m38.489988972s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.620119909 (554h10m38.491202954s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.624843093 (554h10m38.495926158s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.628861344 (554h10m38.499944389s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635131774 (554h10m38.506214809s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635396528 (554h10m38.506479563s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.872118620 (554h10m38.743201655s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.888451481 (554h10m38.759534526s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997493587 (554h10m38.868576632s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997518144 (554h10m38.868601189s) [seg_pos={0 0}, upload=485859]: Segment reached optimal threshold of 13 pieces. Dec 28 17:39:38.997763581 (554h10m38.868846626s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997773039 (554h10m38.868856084s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.997910611 (554h10m38.868993646s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997920089 (554h10m38.869003124s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998111073 (554h10m38.869194128s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.998121602 (554h10m38.869204648s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998151700 (554h10m38.869234735s) [seg_pos={0 0}, upload=485859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:39:38.998234527 (554h10m38.869317572s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:39:38.998242272 (554h10m38.869325317s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:39:39.301480563 (554h10m39.172563608s) [upload=485859]: Done uploading segments... Dec 28 17:39:39.301499209 (554h10m39.172582254s) [upload=485859]: Upload finished. err: <nil> Duration: 6.22423033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 47.973668507s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:44:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 47.973698995s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....X..X....................X..X.............X..............] duration: 3.561413957s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:43:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.561448243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..XX.........XXX...XX...X.X..XXX.XXX...XX...XX.X..XX.......] duration: 6.075906454s trace: https://jaeger.dev.storj.io/trace/3a541c3d76227692 FAIL (2025-12-28 17:43:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A) context deadline exceeded ======LOGS====== Dec 28 17:39:36.354400600 (554h10m36.225483635s) [upload=485859]: Starting upload Dec 28 17:39:36.354469531 (554h10m36.225552576s) [upload=485859]: Done starting upload Dec 28 17:39:36.354642701 (554h10m36.225725746s) [upload=485859]: Uploading segments... Dec 28 17:39:36.354914058 (554h10m36.225997093s) [upload=485859]: Got next segment. Inline: false Dec 28 17:39:36.354956999 (554h10m36.226040034s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:39:36.354961368 (554h10m36.226044413s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:39:36.601247130 (554h10m36.472330185s) [seg_pos={0 0}, upload=485859]: Begin upload segment... Dec 28 17:39:36.601335769 (554h10m36.472418804s) [seg_pos={0 0}, upload=485859]: Done begin upload segment. Dec 28 17:39:36.601377288 (554h10m36.472460323s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.601413357 (554h10m36.472496402s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602179746 (554h10m36.473262801s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602519904 (554h10m36.473602949s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603138352 (554h10m36.474221387s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603634567 (554h10m36.474717612s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603905994 (554h10m36.474989049s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604115352 (554h10m36.475198387s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604430973 (554h10m36.475514018s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604675990 (554h10m36.475759025s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604768907 (554h10m36.475851952s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.605244272 (554h10m36.476327317s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.611659136 (554h10m36.482742181s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.613275875 (554h10m36.484358921s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616040870 (554h10m36.487123915s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616651533 (554h10m36.487734568s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:38.318685603 (554h10m38.189768648s) [upload=485859]: Next returned nil segment Dec 28 17:39:38.318702345 (554h10m38.18978538s) [upload=485859]: Waiting for error group managing segments... Dec 28 17:39:38.601154966 (554h10m38.472238001s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.605166786 (554h10m38.476249831s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.608587720 (554h10m38.479670755s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.613615423 (554h10m38.484698468s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.618905937 (554h10m38.489988972s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.620119909 (554h10m38.491202954s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.624843093 (554h10m38.495926158s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.628861344 (554h10m38.499944389s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635131774 (554h10m38.506214809s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635396528 (554h10m38.506479563s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.872118620 (554h10m38.743201655s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.888451481 (554h10m38.759534526s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997493587 (554h10m38.868576632s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997518144 (554h10m38.868601189s) [seg_pos={0 0}, upload=485859]: Segment reached optimal threshold of 13 pieces. Dec 28 17:39:38.997763581 (554h10m38.868846626s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997773039 (554h10m38.868856084s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.997910611 (554h10m38.868993646s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997920089 (554h10m38.869003124s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998111073 (554h10m38.869194128s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.998121602 (554h10m38.869204648s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998151700 (554h10m38.869234735s) [seg_pos={0 0}, upload=485859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:39:38.998234527 (554h10m38.869317572s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:39:38.998242272 (554h10m38.869325317s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:39:39.301480563 (554h10m39.172563608s) [upload=485859]: Done uploading segments... Dec 28 17:39:39.301499209 (554h10m39.172582254s) [upload=485859]: Upload finished. err: <nil> Duration: 6.075930882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX..XX.........XXX...XX...X.X..XXX.XXX...XX...XX.X..XX......] duration: 5.847845075s trace: https://jaeger.dev.storj.io/trace/3a541c3d76227692 FAIL (2025-12-28 17:43:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A) context deadline exceeded ======LOGS====== Dec 28 17:39:36.354400600 (554h10m36.225483635s) [upload=485859]: Starting upload Dec 28 17:39:36.354469531 (554h10m36.225552576s) [upload=485859]: Done starting upload Dec 28 17:39:36.354642701 (554h10m36.225725746s) [upload=485859]: Uploading segments... Dec 28 17:39:36.354914058 (554h10m36.225997093s) [upload=485859]: Got next segment. Inline: false Dec 28 17:39:36.354956999 (554h10m36.226040034s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:39:36.354961368 (554h10m36.226044413s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:39:36.601247130 (554h10m36.472330185s) [seg_pos={0 0}, upload=485859]: Begin upload segment... Dec 28 17:39:36.601335769 (554h10m36.472418804s) [seg_pos={0 0}, upload=485859]: Done begin upload segment. Dec 28 17:39:36.601377288 (554h10m36.472460323s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.601413357 (554h10m36.472496402s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602179746 (554h10m36.473262801s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602519904 (554h10m36.473602949s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603138352 (554h10m36.474221387s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603634567 (554h10m36.474717612s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603905994 (554h10m36.474989049s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604115352 (554h10m36.475198387s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604430973 (554h10m36.475514018s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604675990 (554h10m36.475759025s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604768907 (554h10m36.475851952s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.605244272 (554h10m36.476327317s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.611659136 (554h10m36.482742181s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.613275875 (554h10m36.484358921s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616040870 (554h10m36.487123915s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616651533 (554h10m36.487734568s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:38.318685603 (554h10m38.189768648s) [upload=485859]: Next returned nil segment Dec 28 17:39:38.318702345 (554h10m38.18978538s) [upload=485859]: Waiting for error group managing segments... Dec 28 17:39:38.601154966 (554h10m38.472238001s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.605166786 (554h10m38.476249831s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.608587720 (554h10m38.479670755s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.613615423 (554h10m38.484698468s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.618905937 (554h10m38.489988972s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.620119909 (554h10m38.491202954s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.624843093 (554h10m38.495926158s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.628861344 (554h10m38.499944389s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635131774 (554h10m38.506214809s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635396528 (554h10m38.506479563s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.872118620 (554h10m38.743201655s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.888451481 (554h10m38.759534526s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997493587 (554h10m38.868576632s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997518144 (554h10m38.868601189s) [seg_pos={0 0}, upload=485859]: Segment reached optimal threshold of 13 pieces. Dec 28 17:39:38.997763581 (554h10m38.868846626s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997773039 (554h10m38.868856084s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.997910611 (554h10m38.868993646s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997920089 (554h10m38.869003124s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998111073 (554h10m38.869194128s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.998121602 (554h10m38.869204648s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998151700 (554h10m38.869234735s) [seg_pos={0 0}, upload=485859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:39:38.998234527 (554h10m38.869317572s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:39:38.998242272 (554h10m38.869325317s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:39:39.301480563 (554h10m39.172563608s) [upload=485859]: Done uploading segments... Dec 28 17:39:39.301499209 (554h10m39.172582254s) [upload=485859]: Upload finished. err: <nil> Duration: 5.847873107s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....X..X....................X..X.............X.............] duration: 3.664438614s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:43:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.664463281s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......X..X....................X..X.............X............] duration: 5.001247021s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:42:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001260506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 32.531638479s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:42:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 32.53165517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX..XX.........XXX...XX...X.X..XXX.XXX...XX...XX.X..XX.....] duration: 7.389667021s trace: https://jaeger.dev.storj.io/trace/3a541c3d76227692 FAIL (2025-12-28 17:42:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A) context deadline exceeded ======LOGS====== Dec 28 17:39:36.354400600 (554h10m36.225483635s) [upload=485859]: Starting upload Dec 28 17:39:36.354469531 (554h10m36.225552576s) [upload=485859]: Done starting upload Dec 28 17:39:36.354642701 (554h10m36.225725746s) [upload=485859]: Uploading segments... Dec 28 17:39:36.354914058 (554h10m36.225997093s) [upload=485859]: Got next segment. Inline: false Dec 28 17:39:36.354956999 (554h10m36.226040034s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:39:36.354961368 (554h10m36.226044413s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:39:36.601247130 (554h10m36.472330185s) [seg_pos={0 0}, upload=485859]: Begin upload segment... Dec 28 17:39:36.601335769 (554h10m36.472418804s) [seg_pos={0 0}, upload=485859]: Done begin upload segment. Dec 28 17:39:36.601377288 (554h10m36.472460323s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.601413357 (554h10m36.472496402s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602179746 (554h10m36.473262801s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602519904 (554h10m36.473602949s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603138352 (554h10m36.474221387s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603634567 (554h10m36.474717612s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603905994 (554h10m36.474989049s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604115352 (554h10m36.475198387s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604430973 (554h10m36.475514018s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604675990 (554h10m36.475759025s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604768907 (554h10m36.475851952s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.605244272 (554h10m36.476327317s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.611659136 (554h10m36.482742181s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.613275875 (554h10m36.484358921s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616040870 (554h10m36.487123915s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616651533 (554h10m36.487734568s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:38.318685603 (554h10m38.189768648s) [upload=485859]: Next returned nil segment Dec 28 17:39:38.318702345 (554h10m38.18978538s) [upload=485859]: Waiting for error group managing segments... Dec 28 17:39:38.601154966 (554h10m38.472238001s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.605166786 (554h10m38.476249831s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.608587720 (554h10m38.479670755s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.613615423 (554h10m38.484698468s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.618905937 (554h10m38.489988972s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.620119909 (554h10m38.491202954s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.624843093 (554h10m38.495926158s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.628861344 (554h10m38.499944389s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635131774 (554h10m38.506214809s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635396528 (554h10m38.506479563s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.872118620 (554h10m38.743201655s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.888451481 (554h10m38.759534526s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997493587 (554h10m38.868576632s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997518144 (554h10m38.868601189s) [seg_pos={0 0}, upload=485859]: Segment reached optimal threshold of 13 pieces. Dec 28 17:39:38.997763581 (554h10m38.868846626s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997773039 (554h10m38.868856084s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.997910611 (554h10m38.868993646s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997920089 (554h10m38.869003124s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998111073 (554h10m38.869194128s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.998121602 (554h10m38.869204648s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998151700 (554h10m38.869234735s) [seg_pos={0 0}, upload=485859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:39:38.998234527 (554h10m38.869317572s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:39:38.998242272 (554h10m38.869325317s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:39:39.301480563 (554h10m39.172563608s) [upload=485859]: Done uploading segments... Dec 28 17:39:39.301499209 (554h10m39.172582254s) [upload=485859]: Upload finished. err: <nil> Duration: 7.389694604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XX..XX.........XXX...XX...X.X..XXX.XXX...XX...XX.X..XX....] duration: 5.728063848s trace: https://jaeger.dev.storj.io/trace/3a541c3d76227692 FAIL (2025-12-28 17:42:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A) context deadline exceeded ======LOGS====== Dec 28 17:39:36.354400600 (554h10m36.225483635s) [upload=485859]: Starting upload Dec 28 17:39:36.354469531 (554h10m36.225552576s) [upload=485859]: Done starting upload Dec 28 17:39:36.354642701 (554h10m36.225725746s) [upload=485859]: Uploading segments... Dec 28 17:39:36.354914058 (554h10m36.225997093s) [upload=485859]: Got next segment. Inline: false Dec 28 17:39:36.354956999 (554h10m36.226040034s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:39:36.354961368 (554h10m36.226044413s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:39:36.601247130 (554h10m36.472330185s) [seg_pos={0 0}, upload=485859]: Begin upload segment... Dec 28 17:39:36.601335769 (554h10m36.472418804s) [seg_pos={0 0}, upload=485859]: Done begin upload segment. Dec 28 17:39:36.601377288 (554h10m36.472460323s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.601413357 (554h10m36.472496402s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602179746 (554h10m36.473262801s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602519904 (554h10m36.473602949s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603138352 (554h10m36.474221387s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603634567 (554h10m36.474717612s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603905994 (554h10m36.474989049s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604115352 (554h10m36.475198387s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604430973 (554h10m36.475514018s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604675990 (554h10m36.475759025s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604768907 (554h10m36.475851952s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.605244272 (554h10m36.476327317s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.611659136 (554h10m36.482742181s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.613275875 (554h10m36.484358921s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616040870 (554h10m36.487123915s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616651533 (554h10m36.487734568s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:38.318685603 (554h10m38.189768648s) [upload=485859]: Next returned nil segment Dec 28 17:39:38.318702345 (554h10m38.18978538s) [upload=485859]: Waiting for error group managing segments... Dec 28 17:39:38.601154966 (554h10m38.472238001s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.605166786 (554h10m38.476249831s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.608587720 (554h10m38.479670755s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.613615423 (554h10m38.484698468s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.618905937 (554h10m38.489988972s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.620119909 (554h10m38.491202954s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.624843093 (554h10m38.495926158s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.628861344 (554h10m38.499944389s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635131774 (554h10m38.506214809s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635396528 (554h10m38.506479563s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.872118620 (554h10m38.743201655s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.888451481 (554h10m38.759534526s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997493587 (554h10m38.868576632s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997518144 (554h10m38.868601189s) [seg_pos={0 0}, upload=485859]: Segment reached optimal threshold of 13 pieces. Dec 28 17:39:38.997763581 (554h10m38.868846626s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997773039 (554h10m38.868856084s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.997910611 (554h10m38.868993646s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997920089 (554h10m38.869003124s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998111073 (554h10m38.869194128s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.998121602 (554h10m38.869204648s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998151700 (554h10m38.869234735s) [seg_pos={0 0}, upload=485859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:39:38.998234527 (554h10m38.869317572s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:39:38.998242272 (554h10m38.869325317s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:39:39.301480563 (554h10m39.172563608s) [upload=485859]: Done uploading segments... Dec 28 17:39:39.301499209 (554h10m39.172582254s) [upload=485859]: Upload finished. err: <nil> Duration: 5.728096139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 42.624063648s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:42:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 42.624088516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.......X..X....................X..X.............X...........] duration: 3.381203051s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:41:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.381295448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...XX..XX.........XXX...XX...X.X..XXX.XXX...XX...XX.X..XX...] duration: 5.743814733s trace: https://jaeger.dev.storj.io/trace/3a541c3d76227692 FAIL (2025-12-28 17:41:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A) context deadline exceeded ======LOGS====== Dec 28 17:39:36.354400600 (554h10m36.225483635s) [upload=485859]: Starting upload Dec 28 17:39:36.354469531 (554h10m36.225552576s) [upload=485859]: Done starting upload Dec 28 17:39:36.354642701 (554h10m36.225725746s) [upload=485859]: Uploading segments... Dec 28 17:39:36.354914058 (554h10m36.225997093s) [upload=485859]: Got next segment. Inline: false Dec 28 17:39:36.354956999 (554h10m36.226040034s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:39:36.354961368 (554h10m36.226044413s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:39:36.601247130 (554h10m36.472330185s) [seg_pos={0 0}, upload=485859]: Begin upload segment... Dec 28 17:39:36.601335769 (554h10m36.472418804s) [seg_pos={0 0}, upload=485859]: Done begin upload segment. Dec 28 17:39:36.601377288 (554h10m36.472460323s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.601413357 (554h10m36.472496402s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602179746 (554h10m36.473262801s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.602519904 (554h10m36.473602949s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603138352 (554h10m36.474221387s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603634567 (554h10m36.474717612s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.603905994 (554h10m36.474989049s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604115352 (554h10m36.475198387s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604430973 (554h10m36.475514018s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604675990 (554h10m36.475759025s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.604768907 (554h10m36.475851952s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.605244272 (554h10m36.476327317s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.611659136 (554h10m36.482742181s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.613275875 (554h10m36.484358921s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616040870 (554h10m36.487123915s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:36.616651533 (554h10m36.487734568s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Uploading piece... Dec 28 17:39:38.318685603 (554h10m38.189768648s) [upload=485859]: Next returned nil segment Dec 28 17:39:38.318702345 (554h10m38.18978538s) [upload=485859]: Waiting for error group managing segments... Dec 28 17:39:38.601154966 (554h10m38.472238001s) [noise=true , address=145.79.227.22:4510 , piece_id=LC2GIIDEDACNLQCNQGRSOSIJVF6HOLP7AA53J2RRNIUFVO4POQOA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.605166786 (554h10m38.476249831s) [noise=true , address=145.79.227.3:4151 , piece_id=BWSVFUZUH5NFOCESQRZY5KIVDF3XOQWRCLA3PX2ZO6N2IHS3J7HA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.608587720 (554h10m38.479670755s) [noise=true , address=145.79.227.48:4303 , piece_id=UEEF4MEYIJ7WAQBWJ7ZPCZOC3UGUZSN7S2NCUZZWTZZZS2JTEFCA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.613615423 (554h10m38.484698468s) [noise=true , address=145.79.227.51:4411 , piece_id=MIE7WJFIBNY6YI2LEUD55K6RPLTUMR2CJLDGS5X37PZI2OTEL5NA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.618905937 (554h10m38.489988972s) [noise=true , address=145.79.227.47:4267 , piece_id=VSRS5BAO7Z4NYKJVLD3F7UJEIS76L3TWYESWZ4VMZHZTIUQOHPIA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.620119909 (554h10m38.491202954s) [noise=true , address=145.79.227.50:4374 , piece_id=IGAMF5SIKTVOYT2IETAUCYYBLNFG2XFIHVHIAZS3NUSLAXYNCJXQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.624843093 (554h10m38.495926158s) [noise=true , address=145.79.227.14:4839 , piece_id=IMH6B6JJMIVISIPQO2EHSDZ2PKDSDRGR2WCBZTR5VFZYUHJPHCUA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.628861344 (554h10m38.499944389s) [noise=true , address=145.79.227.21:4473 , piece_id=SOXV6PQ3QKPWW5N5CFNW2BF2P2AV6FAT3HSJBRMZLUGMAE6HG6RA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635131774 (554h10m38.506214809s) [noise=true , address=145.79.227.42:4122 , piece_id=LOQGZKCAXYK5G2SIEGSXQRLA3Z6SEKIJMEBE5UWR5DAKHGHN37RQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.635396528 (554h10m38.506479563s) [noise=true , address=145.79.227.2:4115 , piece_id=6YAI42FPQLC3XAOFTXCXE5LDWXDYXZ2K6ZZBLLU5ZVSB7U75JFTQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.872118620 (554h10m38.743201655s) [noise=true , address=77.90.31.11:4555 , piece_id=QBWJA7FUUPZNTX2XBABZ3INWRLIZUQTAGPTOYOD4NBOOOPJANQ7A, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.888451481 (554h10m38.759534526s) [noise=true , address=45.91.116.8:4047 , piece_id=YQBKDWUX3TNFHPGEDBHGIPEU7LKOLWCRJNARGZTLHWMO7SFZT4BQ, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997493587 (554h10m38.868576632s) [noise=true , address=45.91.116.10:4084 , piece_id=P4MUIHJBVSCCTMU227NJEBZZRZOYW6I4NGOYJZAMORVIC33PBATA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: <nil> Dec 28 17:39:38.997518144 (554h10m38.868601189s) [seg_pos={0 0}, upload=485859]: Segment reached optimal threshold of 13 pieces. Dec 28 17:39:38.997763581 (554h10m38.868846626s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997773039 (554h10m38.868856084s) [noise=true , address=77.90.31.14:4664 , piece_id=MQSHIHZZR6E4QPPVPDJL6LP4XTSIXGY6FD7K6Y6MTRRCNN4NYEKA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.997910611 (554h10m38.868993646s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.997920089 (554h10m38.869003124s) [noise=true , address=145.79.227.35:4483 , piece_id=AN4BZ3FB3IG3BI7NNT2SZVXZSVQNOG6Q63H7CLHN4XBPUFBUEHBA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998111073 (554h10m38.869194128s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 17:39:38.998121602 (554h10m38.869204648s) [noise=true , address=77.90.31.10:4517 , piece_id=Y6YCVPB735JEMMCWVMXCCC3WEZVKTDFBEWE7ERP6TZUNPIHHXJQA, seg_pos={0 0}, upload=485859]: Long tail cancellation due to optimal threshold Dec 28 17:39:38.998151700 (554h10m38.869234735s) [seg_pos={0 0}, upload=485859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:39:38.998234527 (554h10m38.869317572s) [upload=485859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:39:38.998242272 (554h10m38.869325317s) [upload=485859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:39:39.301480563 (554h10m39.172563608s) [upload=485859]: Done uploading segments... Dec 28 17:39:39.301499209 (554h10m39.172582254s) [upload=485859]: Upload finished. err: <nil> Duration: 5.74384995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 38.456109731s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:40:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 38.456139899s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X.......X..X....................X..X.............X..........] duration: 4.496057826s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:40:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.4962333s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)