Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.XX] duration: 5.000577234s trace: https://jaeger.dev.storj.io/trace/f1e867a35f88a4b FAIL (2025-12-29 16:44:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000598224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.......X.......] duration: 51.000079826s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:44:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 51.000095666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [....X.....X......X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXXX] duration: 30.003255913s trace: https://jaeger.dev.storj.io/trace/2f8216c679c15bf5 FAIL (2025-12-29 16:44:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: BC6H3MK5CE7LCO6CYP7MTFWZD446FC3GHCXWMWIHIKRMIPGMYIMQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VWF5RCFLPFUVZR7NUQYEPTPFG7U3U3FWFAYEARHNJSU4ATSSHZEA) context deadline exceeded; piecestore: (Node ID: 12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, Piece ID: FIKMVZ46MT4KYQLFE3IBCB7U7A5ARJQ6EIK74L4BJWDUACCU42LA) context deadline exceeded ======LOGS====== ec 29 16:43:57.079165072 (577h14m56.950248107s) [upload=506038]: Starting upload Dec 29 16:43:57.079284540 (577h14m56.950367585s) [upload=506038]: Done starting upload Dec 29 16:43:57.079345285 (577h14m56.95042834s) [upload=506038]: Uploading segments... Dec 29 16:43:57.079556147 (577h14m56.950639192s) [upload=506038]: Got next segment. Inline: false Dec 29 16:43:57.079598358 (577h14m56.950681393s) [upload=506038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:43:57.079604359 (577h14m56.950687404s) [upload=506038]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:43:57.324822381 (577h14m57.195905436s) [seg_pos={0 0}, upload=506038]: Begin upload segment... Dec 29 16:43:57.324879369 (577h14m57.195962414s) [seg_pos={0 0}, upload=506038]: Done begin upload segment. Dec 29 16:43:57.325071175 (577h14m57.19615422s) [noise=true , address=45.91.116.10:4081 , piece_id=VWF5RCFLPFUVZR7NUQYEPTPFG7U3U3FWFAYEARHNJSU4ATSSHZEA, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.325119687 (577h14m57.196202722s) [noise=true , address=145.79.227.26:4653 , piece_id=NOVMGYWJO4NYGFSN2LJGKYCTCW3RLIYIFNKQRPI2ULWBDHA4TROA, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.325644095 (577h14m57.19672714s) [noise=true , address=145.79.227.39:4014 , piece_id=FVO3NZ347LATRPALSX3LGCCTIIM7SMEUK5CVEOW6VND4RZTNCOVA, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.326105914 (577h14m57.197188949s) [noise=true , address=145.79.227.13:4804 , piece_id=FIKMVZ46MT4KYQLFE3IBCB7U7A5ARJQ6EIK74L4BJWDUACCU42LA, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.326509583 (577h14m57.197592618s) [noise=true , address=45.91.116.13:4155 , piece_id=BC6H3MK5CE7LCO6CYP7MTFWZD446FC3GHCXWMWIHIKRMIPGMYIMQ, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.326912832 (577h14m57.197995867s) [noise=true , address=145.79.227.51:4411 , piece_id=R4U3XZNVIKBVJOT2GYQOQDUD7Y7D7EK3DWWIV42UI4WBH73MWKOQ, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.327337871 (577h14m57.198420916s) [noise=true , address=145.79.227.19:4079 , piece_id=3UES4TSEVAESUHXPJKPKJ65L7AX67B2ARXZUJPLBOFUR3GQQOHTQ, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.327638272 (577h14m57.198721307s) [noise=true , address=145.79.227.12:4765 , piece_id=62EF7DNP7DOZALGXMSRMFNZOLYES7F7GHRSNBTOETWHS6LECUVGA, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.327983349 (577h14m57.199066384s) [noise=true , address=145.79.227.41:4085 , piece_id=M7QEBZRV4T7YYSRNM2HRPQLB6RTSRBN3JPZ245FCZ5E65EIS2XLA, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.329337318 (577h14m57.200420353s) [noise=true , address=145.79.227.40:4051 , piece_id=TMYUDAJNS2WJDYPLDG636GAA4FBCWWSLAWSI7DIOR36FPJVBM3GA, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.329435735 (577h14m57.20051878s) [noise=true , address=45.91.116.4:4937 , piece_id=ECMXDMY2ANMH7XD76ANAVRZ6QNDEEDMR6K23FOPVBD7AQUYBKGUA, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.330700453 (577h14m57.201783488s) [noise=true , address=145.79.227.22:4509 , piece_id=QLNHJP7UXOUJLZCR6Y3LA2WJCPJRICNFTFH72UDNY5EYJVOEZASA, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.330923879 (577h14m57.202006924s) [noise=true , address=45.91.116.12:4225 , piece_id=K7UCSUUPUJBUWYKCZQUOU7FZSG5HG2HS63KPVWCG7SKIL57JVVNA, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.332342009 (577h14m57.203425044s) [noise=true , address=145.79.227.37:4567 , piece_id=GNAAY3HWXOFMY4LKZVB4YJ25H3THZ2RJIMYHTEH4WA6CNMFXJYMA, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.334220907 (577h14m57.205303952s) [noise=true , address=77.90.31.14:4663 , piece_id=6EAVF5OP5J7CPAY3Q4VZIW26XP7DFFGSFGYISKUTKGQPBTQ4X26A, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:57.335491847 (577h14m57.206574882s) [noise=true , address=145.79.227.15:4875 , piece_id=7BXXCRE4JISORHXDHHYYFFGNWUX4Z6TAZJ4WDDZSEZVPEVICCFHQ, seg_pos={0 0}, upload=506038]: Uploading piece... Dec 29 16:43:59.034427032 (577h14m58.905510067s) [upload=506038]: Next returned nil segment Dec 29 16:43:59.034439516 (577h14m58.905522551s) [upload=506038]: Waiting for error group managing segments... Dec 29 16:43:59.310306958 (577h14m59.181389993s) [noise=true , address=145.79.227.51:4411 , piece_id=R4U3XZNVIKBVJOT2GYQOQDUD7Y7D7EK3DWWIV42UI4WBH73MWKOQ, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.313659522 (577h14m59.184742567s) [noise=true , address=145.79.227.15:4875 , piece_id=7BXXCRE4JISORHXDHHYYFFGNWUX4Z6TAZJ4WDDZSEZVPEVICCFHQ, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.317390746 (577h14m59.188473781s) [noise=true , address=145.79.227.37:4567 , piece_id=GNAAY3HWXOFMY4LKZVB4YJ25H3THZ2RJIMYHTEH4WA6CNMFXJYMA, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.334405515 (577h14m59.20548856s) [noise=true , address=145.79.227.22:4509 , piece_id=QLNHJP7UXOUJLZCR6Y3LA2WJCPJRICNFTFH72UDNY5EYJVOEZASA, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.334543447 (577h14m59.205626482s) [noise=true , address=145.79.227.26:4653 , piece_id=NOVMGYWJO4NYGFSN2LJGKYCTCW3RLIYIFNKQRPI2ULWBDHA4TROA, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.335573509 (577h14m59.206656544s) [noise=true , address=145.79.227.39:4014 , piece_id=FVO3NZ347LATRPALSX3LGCCTIIM7SMEUK5CVEOW6VND4RZTNCOVA, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.338686416 (577h14m59.209769451s) [noise=true , address=145.79.227.41:4085 , piece_id=M7QEBZRV4T7YYSRNM2HRPQLB6RTSRBN3JPZ245FCZ5E65EIS2XLA, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.340278448 (577h14m59.211361483s) [noise=true , address=145.79.227.19:4079 , piece_id=3UES4TSEVAESUHXPJKPKJ65L7AX67B2ARXZUJPLBOFUR3GQQOHTQ, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.429416741 (577h14m59.300499786s) [noise=true , address=145.79.227.12:4765 , piece_id=62EF7DNP7DOZALGXMSRMFNZOLYES7F7GHRSNBTOETWHS6LECUVGA, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.542897956 (577h14m59.413981001s) [noise=true , address=45.91.116.13:4155 , piece_id=BC6H3MK5CE7LCO6CYP7MTFWZD446FC3GHCXWMWIHIKRMIPGMYIMQ, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.581976588 (577h14m59.453059633s) [noise=true , address=45.91.116.10:4081 , piece_id=VWF5RCFLPFUVZR7NUQYEPTPFG7U3U3FWFAYEARHNJSU4ATSSHZEA, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.672440464 (577h14m59.543523499s) [noise=true , address=45.91.116.4:4937 , piece_id=ECMXDMY2ANMH7XD76ANAVRZ6QNDEEDMR6K23FOPVBD7AQUYBKGUA, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.694075981 (577h14m59.565159016s) [noise=true , address=145.79.227.13:4804 , piece_id=FIKMVZ46MT4KYQLFE3IBCB7U7A5ARJQ6EIK74L4BJWDUACCU42LA, seg_pos={0 0}, upload=506038]: Done uploading piece. err: <nil> Dec 29 16:43:59.694100839 (577h14m59.565183894s) [seg_pos={0 0}, upload=506038]: Segment reached optimal threshold of 13 pieces. Dec 29 16:43:59.694469360 (577h14m59.565552395s) [noise=true , address=45.91.116.12:4225 , piece_id=K7UCSUUPUJBUWYKCZQUOU7FZSG5HG2HS63KPVWCG7SKIL57JVVNA, seg_pos={0 0}, upload=506038]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 16:43:59.694499127 (577h14m59.565582172s) [noise=true , address=45.91.116.12:4225 , piece_id=K7UCSUUPUJBUWYKCZQUOU7FZSG5HG2HS63KPVWCG7SKIL57JVVNA, seg_pos={0 0}, upload=506038]: Long tail cancellation due to optimal threshold Dec 29 16:43:59.694512954 (577h14m59.565595999s) [noise=true , address=145.79.227.40:4051 , piece_id=TMYUDAJNS2WJDYPLDG636GAA4FBCWWSLAWSI7DIOR36FPJVBM3GA, seg_pos={0 0}, upload=506038]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 16:43:59.694519195 (577h14m59.56560223s) [noise=true , address=145.79.227.40:4051 , piece_id=TMYUDAJNS2WJDYPLDG636GAA4FBCWWSLAWSI7DIOR36FPJVBM3GA, seg_pos={0 0}, upload=506038]: Long tail cancellation due to optimal threshold Dec 29 16:43:59.694596011 (577h14m59.565679056s) [noise=true , address=77.90.31.14:4663 , piece_id=6EAVF5OP5J7CPAY3Q4VZIW26XP7DFFGSFGYISKUTKGQPBTQ4X26A, seg_pos={0 0}, upload=506038]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:42254->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:42254->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42254->77.90.31.14:4663: use of closed network connection Dec 29 16:43:59.694606341 (577h14m59.565689376s) [noise=true , address=77.90.31.14:4663 , piece_id=6EAVF5OP5J7CPAY3Q4VZIW26XP7DFFGSFGYISKUTKGQPBTQ4X26A, seg_pos={0 0}, upload=506038]: Long tail cancellation due to optimal threshold Dec 29 16:43:59.694631379 (577h14m59.565714414s) [seg_pos={0 0}, upload=506038]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:43:59.694749283 (577h14m59.565832318s) [upload=506038]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:43:59.694757078 (577h14m59.565840113s) [upload=506038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:43:59.987587691 (577h14m59.858670736s) [upload=506038]: Done uploading segments... Dec 29 16:43:59.987610285 (577h14m59.85869332s) [upload=506038]: Upload finished. err: <nil> Duration: 30.003282143s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.X] duration: 5.000902779s trace: https://jaeger.dev.storj.io/trace/173f8370141a110f FAIL (2025-12-29 16:44:20 UTC) orbiter warning: context deadline exceeded Duration: 5.000923197s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X.] duration: 4.438815744s trace: https://jaeger.dev.storj.io/trace/10ed21dbb9f0820b FAIL (2025-12-29 16:43:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.438871211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.......X......] duration: 23.194623131s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:43:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 23.19463882s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.......X.....] duration: 38.160187366s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:43:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 38.160207984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.X] duration: 5.001326349s trace: https://jaeger.dev.storj.io/trace/10ed21dbb9f0820b FAIL (2025-12-29 16:43:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001350063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....X.....X......X...XX.XXX....X...XXXXX.X..XX...XX..XXXXXX] duration: 30.003177477s trace: https://jaeger.dev.storj.io/trace/5a7ad0ad9947e292 FAIL (2025-12-29 16:43:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: B5QEJRVHVTIQFJGGYGSIHPG2WOSVPUI4AX6NOYOTX6DYXZ7JBCKA) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: UOBRYLNSIFKIRZSERRCPGKZRNUQ6AYCQ4OK2OLS7PZA2YLJVQU5Q) context deadline exceeded ======LOGS====== 16:42:43.601731379 (577h13m43.472814414s) [upload=506022]: Starting upload Dec 29 16:42:43.601782576 (577h13m43.472865611s) [upload=506022]: Done starting upload Dec 29 16:42:43.601919116 (577h13m43.473002152s) [upload=506022]: Uploading segments... Dec 29 16:42:43.602127133 (577h13m43.473210168s) [upload=506022]: Got next segment. Inline: false Dec 29 16:42:43.602177989 (577h13m43.473261034s) [upload=506022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:42:43.602182739 (577h13m43.473265784s) [upload=506022]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:42:43.852098004 (577h13m43.723181039s) [seg_pos={0 0}, upload=506022]: Begin upload segment... Dec 29 16:42:43.852142950 (577h13m43.723225985s) [seg_pos={0 0}, upload=506022]: Done begin upload segment. Dec 29 16:42:43.852171334 (577h13m43.723254379s) [noise=true , address=45.91.116.9:4261 , piece_id=B5QEJRVHVTIQFJGGYGSIHPG2WOSVPUI4AX6NOYOTX6DYXZ7JBCKA, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.852310228 (577h13m43.723393294s) [noise=true , address=145.79.227.19:4077 , piece_id=6YOI5M6DTZWX6ECCU6F2CNHNRCU5F2UXRZ327GLZIHELCRSCJOVQ, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.852932323 (577h13m43.724015358s) [noise=true , address=145.79.227.22:4509 , piece_id=IKGEDOUPPZ3XH6MCQ2YUOTBSA2YD3DWHZ6HCDPLFIWADJEIQEM2Q, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.853127846 (577h13m43.724210881s) [noise=true , address=145.79.227.17:4972 , piece_id=EO7FF656NN25X42MRTXRU7JGDSEGEEYJJHBEAGCDHFQM6VH3Q2TQ, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.853556161 (577h13m43.724639196s) [noise=true , address=145.79.227.50:4375 , piece_id=ABQMCJL5UNFRAO6QSBA3UU3CABXZKEXTG7CZ5OR3I5YV6TKZVYHA, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.853579827 (577h13m43.724662872s) [noise=true , address=145.79.227.24:4582 , piece_id=JRX2XUWMRD7A7TO2B6TLSGLVGBHDE7LXMDYRGSFT4NBKSQBDH57Q, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.853886890 (577h13m43.724969925s) [noise=true , address=45.91.116.11:4119 , piece_id=OLR5SSMFOMNRMPV2PCTZ3FOTSSY7HMYU5D47DUSBFHPV6NGNCW3A, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.854187163 (577h13m43.725270198s) [noise=true , address=77.90.31.11:4553 , piece_id=UOBRYLNSIFKIRZSERRCPGKZRNUQ6AYCQ4OK2OLS7PZA2YLJVQU5Q, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.854258599 (577h13m43.725341654s) [noise=true , address=145.79.227.39:4013 , piece_id=LJJQSPQZSNCW5PBRRQHZQR4VLWJLQM34HUPEYQFBFGTAZABEC6LA, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.854503986 (577h13m43.725587031s) [noise=true , address=145.79.227.2:4115 , piece_id=KACCCQJGTYA2NNQW5AOMLXEUOGQP4EK3GE36KTL4VICYT2FD7MYA, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.854826530 (577h13m43.725909575s) [noise=true , address=77.90.31.14:4662 , piece_id=JCY3XDK2URTNVVWZKWT6XTXNJARICJJ2SYHXUVQADWREACNZGHTA, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.855598590 (577h13m43.726681625s) [noise=true , address=145.79.227.37:4565 , piece_id=F6IYLM6SSI62VY4T7FII4KFTCKWJQBEXP5G25QDODQFVMGWNPGZA, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.855811245 (577h13m43.72689428s) [noise=true , address=145.79.227.51:4409 , piece_id=DSFRMZ24VZDTKRR5SAT36ZVO5RPHBXTJXJSPDITTPZKFMV5BDY5Q, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.856272403 (577h13m43.727355438s) [noise=true , address=77.90.31.15:4697 , piece_id=A5AJCSY7NU3TETP5B4YLQYEJL7ONKXCWAD7KAQM2SWGDG5PPGENQ, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.856857126 (577h13m43.727940171s) [noise=true , address=145.79.227.6:4259 , piece_id=SUHXNJG3I6XCAXHO3TJKP73FPSB3BNDNEMNMFXVPVFCSE5FQPRDQ, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:43.858080585 (577h13m43.729163641s) [noise=true , address=145.79.227.9:4366 , piece_id=LIXFEVIP2JKWQITJSAQF5NBEJMN26OQT6IIUTSJZBN25NIHPOGXQ, seg_pos={0 0}, upload=506022]: Uploading piece... Dec 29 16:42:45.544880363 (577h13m45.415963398s) [upload=506022]: Next returned nil segment Dec 29 16:42:45.544893237 (577h13m45.415976282s) [upload=506022]: Waiting for error group managing segments... Dec 29 16:42:45.833072644 (577h13m45.704155679s) [noise=true , address=145.79.227.39:4013 , piece_id=LJJQSPQZSNCW5PBRRQHZQR4VLWJLQM34HUPEYQFBFGTAZABEC6LA, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:45.836391505 (577h13m45.70747455s) [noise=true , address=145.79.227.9:4366 , piece_id=LIXFEVIP2JKWQITJSAQF5NBEJMN26OQT6IIUTSJZBN25NIHPOGXQ, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:45.837222577 (577h13m45.708305622s) [noise=true , address=145.79.227.50:4375 , piece_id=ABQMCJL5UNFRAO6QSBA3UU3CABXZKEXTG7CZ5OR3I5YV6TKZVYHA, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:45.838203024 (577h13m45.709286059s) [noise=true , address=145.79.227.51:4409 , piece_id=DSFRMZ24VZDTKRR5SAT36ZVO5RPHBXTJXJSPDITTPZKFMV5BDY5Q, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:45.844216304 (577h13m45.715299349s) [noise=true , address=145.79.227.22:4509 , piece_id=IKGEDOUPPZ3XH6MCQ2YUOTBSA2YD3DWHZ6HCDPLFIWADJEIQEM2Q, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:45.846037561 (577h13m45.717120596s) [noise=true , address=145.79.227.19:4077 , piece_id=6YOI5M6DTZWX6ECCU6F2CNHNRCU5F2UXRZ327GLZIHELCRSCJOVQ, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:45.855067036 (577h13m45.726150101s) [noise=true , address=145.79.227.2:4115 , piece_id=KACCCQJGTYA2NNQW5AOMLXEUOGQP4EK3GE36KTL4VICYT2FD7MYA, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:45.858362270 (577h13m45.729445305s) [noise=true , address=145.79.227.17:4972 , piece_id=EO7FF656NN25X42MRTXRU7JGDSEGEEYJJHBEAGCDHFQM6VH3Q2TQ, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:45.858450948 (577h13m45.729533983s) [noise=true , address=145.79.227.24:4582 , piece_id=JRX2XUWMRD7A7TO2B6TLSGLVGBHDE7LXMDYRGSFT4NBKSQBDH57Q, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:45.873190796 (577h13m45.744273831s) [noise=true , address=145.79.227.6:4259 , piece_id=SUHXNJG3I6XCAXHO3TJKP73FPSB3BNDNEMNMFXVPVFCSE5FQPRDQ, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:45.876899848 (577h13m45.747982883s) [noise=true , address=145.79.227.37:4565 , piece_id=F6IYLM6SSI62VY4T7FII4KFTCKWJQBEXP5G25QDODQFVMGWNPGZA, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:46.130881106 (577h13m46.001964151s) [noise=true , address=77.90.31.11:4553 , piece_id=UOBRYLNSIFKIRZSERRCPGKZRNUQ6AYCQ4OK2OLS7PZA2YLJVQU5Q, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:46.134228189 (577h13m46.005311224s) [noise=true , address=45.91.116.9:4261 , piece_id=B5QEJRVHVTIQFJGGYGSIHPG2WOSVPUI4AX6NOYOTX6DYXZ7JBCKA, seg_pos={0 0}, upload=506022]: Done uploading piece. err: <nil> Dec 29 16:42:46.134242327 (577h13m46.005325362s) [seg_pos={0 0}, upload=506022]: Segment reached optimal threshold of 13 pieces. Dec 29 16:42:46.134515617 (577h13m46.005598652s) [noise=true , address=45.91.116.11:4119 , piece_id=OLR5SSMFOMNRMPV2PCTZ3FOTSSY7HMYU5D47DUSBFHPV6NGNCW3A, seg_pos={0 0}, upload=506022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:55500->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:55500->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55500->45.91.116.11:4119: use of closed network connection Dec 29 16:42:46.134523893 (577h13m46.005606928s) [noise=true , address=45.91.116.11:4119 , piece_id=OLR5SSMFOMNRMPV2PCTZ3FOTSSY7HMYU5D47DUSBFHPV6NGNCW3A, seg_pos={0 0}, upload=506022]: Long tail cancellation due to optimal threshold Dec 29 16:42:46.134667686 (577h13m46.005750731s) [noise=true , address=77.90.31.15:4697 , piece_id=A5AJCSY7NU3TETP5B4YLQYEJL7ONKXCWAD7KAQM2SWGDG5PPGENQ, seg_pos={0 0}, upload=506022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 29 16:42:46.134688206 (577h13m46.005771251s) [noise=true , address=77.90.31.15:4697 , piece_id=A5AJCSY7NU3TETP5B4YLQYEJL7ONKXCWAD7KAQM2SWGDG5PPGENQ, seg_pos={0 0}, upload=506022]: Long tail cancellation due to optimal threshold Dec 29 16:42:46.134865253 (577h13m46.005948298s) [noise=true , address=77.90.31.14:4662 , piece_id=JCY3XDK2URTNVVWZKWT6XTXNJARICJJ2SYHXUVQADWREACNZGHTA, seg_pos={0 0}, upload=506022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 29 16:42:46.134873799 (577h13m46.005956834s) [noise=true , address=77.90.31.14:4662 , piece_id=JCY3XDK2URTNVVWZKWT6XTXNJARICJJ2SYHXUVQADWREACNZGHTA, seg_pos={0 0}, upload=506022]: Long tail cancellation due to optimal threshold Dec 29 16:42:46.134908144 (577h13m46.005991189s) [seg_pos={0 0}, upload=506022]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:42:46.135073089 (577h13m46.006156124s) [upload=506022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:42:46.135080653 (577h13m46.006163688s) [upload=506022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:42:46.423196327 (577h13m46.294279372s) [upload=506022]: Done uploading segments... Dec 29 16:42:46.423221335 (577h13m46.29430438s) [upload=506022]: Upload finished. err: <nil> Duration: 30.003192285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.......X....] duration: 34.140862077s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:42:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 34.140877937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [......X.....X......X...XX.XXX....X...XXXXX.X..XX...XX..XXXXX] duration: 30.005329377s trace: https://jaeger.dev.storj.io/trace/2099bc4996edfdd4 FAIL (2025-12-29 16:42:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: AK535YJ64FNL43FNHFVFNYPCFWRE7MYJZNPAELS75RYQMGG5WNFQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: EPQ5EBSCIF3YAHXTBGMCEURRWDP3WYNVK33IGVRWGOZ5VPXJBRCA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: CPW33VU4LTOWNOZ55CKFEZ2O5XQJAXONFWFVWONT6ZDRT62VOCEQ) context deadline exceeded ======LOGS====== 16:42:13.356547092 (577h13m13.227630127s) [upload=506013]: Starting upload Dec 29 16:42:13.356619810 (577h13m13.227702846s) [upload=506013]: Done starting upload Dec 29 16:42:13.356659005 (577h13m13.22774205s) [upload=506013]: Uploading segments... Dec 29 16:42:13.357198933 (577h13m13.228281968s) [upload=506013]: Got next segment. Inline: false Dec 29 16:42:13.357244260 (577h13m13.228327295s) [upload=506013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:42:13.357249149 (577h13m13.228332184s) [upload=506013]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:42:13.607131543 (577h13m13.478214578s) [seg_pos={0 0}, upload=506013]: Begin upload segment... Dec 29 16:42:13.607191657 (577h13m13.478274692s) [seg_pos={0 0}, upload=506013]: Done begin upload segment. Dec 29 16:42:13.607220042 (577h13m13.478303087s) [noise=true , address=145.79.227.19:4077 , piece_id=CH6JM6OMVGTFPEOB7F4WQ4YZCYUENGC5J4DMM6LWNAYTKGC7HCQA, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.607351601 (577h13m13.478434677s) [noise=true , address=145.79.227.46:4601 , piece_id=2OLUY4PCSS2SYNB55QXYZ776JKVTEV7GLGPDKYIX2QJ23R45K4LQ, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.607812650 (577h13m13.478895695s) [noise=true , address=45.91.116.8:4048 , piece_id=7Z3F35KCVPLI5O6YIYJ72CMQXFMOABLKGIXMYDWNY7LZAQ6TDLVQ, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.608238541 (577h13m13.479321576s) [noise=true , address=77.90.31.13:4629 , piece_id=22LE7RJ6IXJPIOHFS3QZBCS67BWCYLU2TNYZ35UEIOZRH2G7E5XA, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.608525226 (577h13m13.479608262s) [noise=true , address=145.79.227.13:4803 , piece_id=ATCL5JY5XMST7ZLPSZWDGSMHOFF35JDQHJXCEWTVJPCTCYP3KJPQ, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.608826781 (577h13m13.479909816s) [noise=true , address=77.90.31.21:4877 , piece_id=EPQ5EBSCIF3YAHXTBGMCEURRWDP3WYNVK33IGVRWGOZ5VPXJBRCA, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.609264113 (577h13m13.480347148s) [noise=true , address=145.79.227.26:4653 , piece_id=FJCWBSKNBU3MGODA7XMGRFHHAQC7E4ZFAZJHFBV6OCYE5DOX3XMA, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.609524209 (577h13m13.480607244s) [noise=true , address=145.79.227.41:4086 , piece_id=AMNX4HLBZNVKHZCQHFN7G3VRYJTLD2RVZ67OTCUVL2B73EHIRDTQ, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.609927647 (577h13m13.481010692s) [noise=true , address=77.90.31.14:4663 , piece_id=WPN42ZRUSRJYBWZLC3JF55SBK5JELQAMMAV7UATRINF67UH3AQNQ, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.610068765 (577h13m13.4811518s) [noise=true , address=145.79.227.22:4509 , piece_id=IWK7JTZI35ZXRLE6QOAL7JKPEBH2MORKKUQOWVOTEJ6SP4JSYPLA, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.610455191 (577h13m13.481538226s) [noise=true , address=145.79.227.51:4411 , piece_id=BU4AYVSSUTQE3S7BOMMC4TCEE6YOZJDNZWR7B7USDRUQPWQP3BSQ, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.610804356 (577h13m13.481887401s) [noise=true , address=45.91.116.7:4011 , piece_id=CPW33VU4LTOWNOZ55CKFEZ2O5XQJAXONFWFVWONT6ZDRT62VOCEQ, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.611113154 (577h13m13.482196199s) [noise=true , address=45.91.116.10:4081 , piece_id=AK535YJ64FNL43FNHFVFNYPCFWRE7MYJZNPAELS75RYQMGG5WNFQ, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.611407144 (577h13m13.482490189s) [noise=true , address=145.79.227.6:4257 , piece_id=JK3DW2Y4OBZ55OJZTPTEPDLMG4ZHUVBLUEYFPGOJ25RQWGVORVOQ, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.611680565 (577h13m13.4827636s) [noise=true , address=145.79.227.50:4375 , piece_id=R4G4JVISHRUE3ACXY644IVK46ZUUBK3B3UBGO4PE7MC6TVRDN7TQ, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:13.612027475 (577h13m13.48311052s) [noise=true , address=45.91.116.9:4262 , piece_id=XWPSAMXYV37UOHONM6ZXKJ7MTOCQOSCXRJ2BBWMNAWI46VCOGBSQ, seg_pos={0 0}, upload=506013]: Uploading piece... Dec 29 16:42:15.305888467 (577h13m15.176971512s) [upload=506013]: Next returned nil segment Dec 29 16:42:15.305903345 (577h13m15.17698638s) [upload=506013]: Waiting for error group managing segments... Dec 29 16:42:15.593376377 (577h13m15.464459423s) [noise=true , address=145.79.227.51:4411 , piece_id=BU4AYVSSUTQE3S7BOMMC4TCEE6YOZJDNZWR7B7USDRUQPWQP3BSQ, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:15.594118711 (577h13m15.465201746s) [noise=true , address=145.79.227.50:4375 , piece_id=R4G4JVISHRUE3ACXY644IVK46ZUUBK3B3UBGO4PE7MC6TVRDN7TQ, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:15.594979370 (577h13m15.466062415s) [noise=true , address=145.79.227.6:4257 , piece_id=JK3DW2Y4OBZ55OJZTPTEPDLMG4ZHUVBLUEYFPGOJ25RQWGVORVOQ, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:15.609075190 (577h13m15.480158225s) [noise=true , address=145.79.227.22:4509 , piece_id=IWK7JTZI35ZXRLE6QOAL7JKPEBH2MORKKUQOWVOTEJ6SP4JSYPLA, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:15.613371752 (577h13m15.484454787s) [noise=true , address=145.79.227.13:4803 , piece_id=ATCL5JY5XMST7ZLPSZWDGSMHOFF35JDQHJXCEWTVJPCTCYP3KJPQ, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:15.615626985 (577h13m15.48671003s) [noise=true , address=145.79.227.41:4086 , piece_id=AMNX4HLBZNVKHZCQHFN7G3VRYJTLD2RVZ67OTCUVL2B73EHIRDTQ, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:15.629879193 (577h13m15.500962229s) [noise=true , address=145.79.227.46:4601 , piece_id=2OLUY4PCSS2SYNB55QXYZ776JKVTEV7GLGPDKYIX2QJ23R45K4LQ, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:15.635595067 (577h13m15.506678102s) [noise=true , address=145.79.227.26:4653 , piece_id=FJCWBSKNBU3MGODA7XMGRFHHAQC7E4ZFAZJHFBV6OCYE5DOX3XMA, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:15.789265148 (577h13m15.660348193s) [noise=true , address=77.90.31.21:4877 , piece_id=EPQ5EBSCIF3YAHXTBGMCEURRWDP3WYNVK33IGVRWGOZ5VPXJBRCA, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:15.833727853 (577h13m15.704810888s) [noise=true , address=45.91.116.8:4048 , piece_id=7Z3F35KCVPLI5O6YIYJ72CMQXFMOABLKGIXMYDWNY7LZAQ6TDLVQ, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:15.897151520 (577h13m15.768234555s) [noise=true , address=45.91.116.10:4081 , piece_id=AK535YJ64FNL43FNHFVFNYPCFWRE7MYJZNPAELS75RYQMGG5WNFQ, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:16.275977292 (577h13m16.147060327s) [noise=true , address=45.91.116.9:4262 , piece_id=XWPSAMXYV37UOHONM6ZXKJ7MTOCQOSCXRJ2BBWMNAWI46VCOGBSQ, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:16.515291561 (577h13m16.386374607s) [noise=true , address=45.91.116.7:4011 , piece_id=CPW33VU4LTOWNOZ55CKFEZ2O5XQJAXONFWFVWONT6ZDRT62VOCEQ, seg_pos={0 0}, upload=506013]: Done uploading piece. err: <nil> Dec 29 16:42:16.515311730 (577h13m16.386394765s) [seg_pos={0 0}, upload=506013]: Segment reached optimal threshold of 13 pieces. Dec 29 16:42:16.515485772 (577h13m16.386568807s) [noise=true , address=145.79.227.19:4077 , piece_id=CH6JM6OMVGTFPEOB7F4WQ4YZCYUENGC5J4DMM6LWNAYTKGC7HCQA, seg_pos={0 0}, upload=506013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:40238->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:40238->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40238->145.79.227.19:4077: use of closed network connection Dec 29 16:42:16.515491152 (577h13m16.386574187s) [noise=true , address=145.79.227.19:4077 , piece_id=CH6JM6OMVGTFPEOB7F4WQ4YZCYUENGC5J4DMM6LWNAYTKGC7HCQA, seg_pos={0 0}, upload=506013]: Long tail cancellation due to optimal threshold Dec 29 16:42:16.515715580 (577h13m16.386798625s) [noise=true , address=77.90.31.14:4663 , piece_id=WPN42ZRUSRJYBWZLC3JF55SBK5JELQAMMAV7UATRINF67UH3AQNQ, seg_pos={0 0}, upload=506013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 29 16:42:16.515721661 (577h13m16.386804706s) [noise=true , address=77.90.31.14:4663 , piece_id=WPN42ZRUSRJYBWZLC3JF55SBK5JELQAMMAV7UATRINF67UH3AQNQ, seg_pos={0 0}, upload=506013]: Long tail cancellation due to optimal threshold Dec 29 16:42:16.515817744 (577h13m16.386900779s) [noise=true , address=77.90.31.13:4629 , piece_id=22LE7RJ6IXJPIOHFS3QZBCS67BWCYLU2TNYZ35UEIOZRH2G7E5XA, seg_pos={0 0}, upload=506013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 29 16:42:16.515822413 (577h13m16.386905458s) [noise=true , address=77.90.31.13:4629 , piece_id=22LE7RJ6IXJPIOHFS3QZBCS67BWCYLU2TNYZ35UEIOZRH2G7E5XA, seg_pos={0 0}, upload=506013]: Long tail cancellation due to optimal threshold Dec 29 16:42:16.515837181 (577h13m16.386920216s) [seg_pos={0 0}, upload=506013]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:42:16.515903858 (577h13m16.386986903s) [upload=506013]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:42:16.515908256 (577h13m16.386991291s) [upload=506013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:42:16.821788978 (577h13m16.692872043s) [upload=506013]: Done uploading segments... Dec 29 16:42:16.822065714 (577h13m16.693148779s) [upload=506013]: Upload finished. err: <nil> Duration: 30.005353593s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X.] duration: 4.899957939s trace: https://jaeger.dev.storj.io/trace/78df9d2d7ea2b017 FAIL (2025-12-29 16:42:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.899977446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.......X...] duration: 38.493310049s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:42:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 38.493327012s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X......X.....X......X...XX.XXX....X...XXXXX.X..XX...XX..XXXX] duration: 30.003246948s trace: https://jaeger.dev.storj.io/trace/197a5de595a23e91 FAIL (2025-12-29 16:42:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:41:31.314257728 (577h12m31.185340763s) [upload=506004]: Starting upload Dec 29 16:41:31.314369842 (577h12m31.185452877s) [upload=506004]: Done starting upload Dec 29 16:41:31.314438472 (577h12m31.185521507s) [upload=506004]: Uploading segments... Dec 29 16:41:31.314578329 (577h12m31.185661364s) [upload=506004]: Got next segment. Inline: false Dec 29 16:41:31.314614407 (577h12m31.185697442s) [upload=506004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:41:31.314618986 (577h12m31.185702021s) [upload=506004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:41:31.565702486 (577h12m31.436785531s) [seg_pos={0 0}, upload=506004]: Begin upload segment... Dec 29 16:41:31.565758442 (577h12m31.436841487s) [seg_pos={0 0}, upload=506004]: Done begin upload segment. Dec 29 16:41:31.565790303 (577h12m31.436873338s) [noise=true , address=45.91.116.8:4048 , piece_id=TK5OAK4XXQKNHAP7KJKF3JSDMWHEM7X7UUFB6LYHK7NIVH35UI3A, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.565947333 (577h12m31.437030378s) [noise=true , address=77.90.31.16:4736 , piece_id=YNQIBJQ54NKBT6QLPLQORADMDO5VPV4ECFU55ANCSI6DU26FHRQA, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.566434490 (577h12m31.437517535s) [noise=true , address=145.79.227.36:4517 , piece_id=DXNKTMKSD7YC4F6BSQF3FEQFWZO5GHFKOQFIOYURS36YY4ZT6XLQ, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.566833980 (577h12m31.437917026s) [noise=true , address=145.79.227.22:4509 , piece_id=ZXSFO2UQJY42ZCIVDK4OQLSM27ELCYNLWYLUHRCYAQJ75QQBBHBQ, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.567248700 (577h12m31.438331745s) [noise=true , address=145.79.227.18:4006 , piece_id=AKVCHLT5W4IVYHBZC6WZUGEDYTPMJV6WV3TAOTLP5HS7GRS3BSWQ, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.567649082 (577h12m31.438732127s) [noise=true , address=145.79.227.16:4909 , piece_id=RFNNHYCF7L5ILP4CU73X2YEF4B527ZCXMHOPA7U2QCUX7SBA5K3A, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.568073561 (577h12m31.439156596s) [noise=true , address=77.90.31.15:4697 , piece_id=UOK35LGVO7BT7KITWLL5HXJM5SHV3XH3XNOGHRYTXET2WTQPMGEA, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.568499011 (577h12m31.439582046s) [noise=true , address=145.79.227.34:4906 , piece_id=FKMBF3ISUYLPF3SEJ7LA5FEVCDDCLFYAA7A5HBIG42QNNHGXN3WA, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.569969832 (577h12m31.441052867s) [noise=true , address=145.79.227.13:4803 , piece_id=DHJ4RODCXSLJFLQPT5JPBU52HVMAKVWBOWANDHF45ZUTOXJZ4FQA, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.570349966 (577h12m31.441433011s) [noise=true , address=77.90.31.10:4518 , piece_id=34URNTWCSFTBQKBRIWVBKEH5XEJ3WOBARL3JFL2PLDWZVSR75USQ, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.571519884 (577h12m31.442602929s) [noise=true , address=45.91.116.10:4081 , piece_id=YBD6GEZPZ7RPLTMWHGI6CEJ3RSMEQAYJXWAQJK4A7QXMHDBNM6XA, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.573086687 (577h12m31.444169732s) [noise=true , address=145.79.227.48:4302 , piece_id=DDXYDLG3X3Q7XAEC6V73RBAUDPEFVQBHQFM6OAXNIZXJ4FZ6LO7A, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.574857169 (577h12m31.445940214s) [noise=true , address=145.79.227.2:4114 , piece_id=EMVA2HPZP42AKYIXWLTRUJ2PRLZHJUEUWQJPQCZBBXZ53XVGSJJA, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.576441805 (577h12m31.44752485s) [noise=true , address=145.79.227.40:4051 , piece_id=SNYBFNDJAWNOYX3RRFHRPTRVRWBPKDTYSPB73V4Y2CCA5E5CAYIQ, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.578045860 (577h12m31.449128905s) [noise=true , address=145.79.227.24:4582 , piece_id=Y7Q3E667KJXEGKJATVM6K2XF2PETZR2IV4RUT4VNEPUZMF6FBJUA, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:31.578986200 (577h12m31.450069245s) [noise=true , address=45.91.116.11:4118 , piece_id=ZETS4ZAMLS3X3VUAP7YH6GXXGTQX4ZBYZYEKDUZLAZ7NQYIKJYUA, seg_pos={0 0}, upload=506004]: Uploading piece... Dec 29 16:41:33.272774605 (577h12m33.14385764s) [upload=506004]: Next returned nil segment Dec 29 16:41:33.272795365 (577h12m33.1438784s) [upload=506004]: Waiting for error group managing segments... Dec 29 16:41:33.556647521 (577h12m33.427730556s) [noise=true , address=145.79.227.36:4517 , piece_id=DXNKTMKSD7YC4F6BSQF3FEQFWZO5GHFKOQFIOYURS36YY4ZT6XLQ, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:33.557751233 (577h12m33.428834278s) [noise=true , address=145.79.227.13:4803 , piece_id=DHJ4RODCXSLJFLQPT5JPBU52HVMAKVWBOWANDHF45ZUTOXJZ4FQA, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:33.558320427 (577h12m33.429403462s) [noise=true , address=145.79.227.22:4509 , piece_id=ZXSFO2UQJY42ZCIVDK4OQLSM27ELCYNLWYLUHRCYAQJ75QQBBHBQ, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:33.577019793 (577h12m33.448102848s) [noise=true , address=145.79.227.2:4114 , piece_id=EMVA2HPZP42AKYIXWLTRUJ2PRLZHJUEUWQJPQCZBBXZ53XVGSJJA, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:33.578443524 (577h12m33.449526569s) [noise=true , address=145.79.227.18:4006 , piece_id=AKVCHLT5W4IVYHBZC6WZUGEDYTPMJV6WV3TAOTLP5HS7GRS3BSWQ, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:33.580395431 (577h12m33.451478476s) [noise=true , address=145.79.227.24:4582 , piece_id=Y7Q3E667KJXEGKJATVM6K2XF2PETZR2IV4RUT4VNEPUZMF6FBJUA, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:33.581066078 (577h12m33.452149113s) [noise=true , address=145.79.227.16:4909 , piece_id=RFNNHYCF7L5ILP4CU73X2YEF4B527ZCXMHOPA7U2QCUX7SBA5K3A, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:33.585639837 (577h12m33.456722882s) [noise=true , address=145.79.227.48:4302 , piece_id=DDXYDLG3X3Q7XAEC6V73RBAUDPEFVQBHQFM6OAXNIZXJ4FZ6LO7A, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:33.835279448 (577h12m33.706362483s) [noise=true , address=77.90.31.16:4736 , piece_id=YNQIBJQ54NKBT6QLPLQORADMDO5VPV4ECFU55ANCSI6DU26FHRQA, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:33.856056721 (577h12m33.727139756s) [noise=true , address=77.90.31.10:4518 , piece_id=34URNTWCSFTBQKBRIWVBKEH5XEJ3WOBARL3JFL2PLDWZVSR75USQ, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:33.856962977 (577h12m33.728046012s) [noise=true , address=45.91.116.8:4048 , piece_id=TK5OAK4XXQKNHAP7KJKF3JSDMWHEM7X7UUFB6LYHK7NIVH35UI3A, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:33.948491540 (577h12m33.819574575s) [noise=true , address=77.90.31.15:4697 , piece_id=UOK35LGVO7BT7KITWLL5HXJM5SHV3XH3XNOGHRYTXET2WTQPMGEA, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:34.649115603 (577h12m34.520198648s) [noise=true , address=45.91.116.11:4118 , piece_id=ZETS4ZAMLS3X3VUAP7YH6GXXGTQX4ZBYZYEKDUZLAZ7NQYIKJYUA, seg_pos={0 0}, upload=506004]: Done uploading piece. err: <nil> Dec 29 16:41:34.649139749 (577h12m34.520222784s) [seg_pos={0 0}, upload=506004]: Segment reached optimal threshold of 13 pieces. Dec 29 16:41:34.649362743 (577h12m34.520445778s) [noise=true , address=145.79.227.40:4051 , piece_id=SNYBFNDJAWNOYX3RRFHRPTRVRWBPKDTYSPB73V4Y2CCA5E5CAYIQ, seg_pos={0 0}, upload=506004]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 16:41:34.649371961 (577h12m34.520455006s) [noise=true , address=145.79.227.40:4051 , piece_id=SNYBFNDJAWNOYX3RRFHRPTRVRWBPKDTYSPB73V4Y2CCA5E5CAYIQ, seg_pos={0 0}, upload=506004]: Long tail cancellation due to optimal threshold Dec 29 16:41:34.649673786 (577h12m34.520756831s) [noise=true , address=145.79.227.34:4906 , piece_id=FKMBF3ISUYLPF3SEJ7LA5FEVCDDCLFYAA7A5HBIG42QNNHGXN3WA, seg_pos={0 0}, upload=506004]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 16:41:34.649682442 (577h12m34.520765477s) [noise=true , address=145.79.227.34:4906 , piece_id=FKMBF3ISUYLPF3SEJ7LA5FEVCDDCLFYAA7A5HBIG42QNNHGXN3WA, seg_pos={0 0}, upload=506004]: Long tail cancellation due to optimal threshold Dec 29 16:41:34.649877885 (577h12m34.52096093s) [noise=true , address=45.91.116.10:4081 , piece_id=YBD6GEZPZ7RPLTMWHGI6CEJ3RSMEQAYJXWAQJK4A7QXMHDBNM6XA, seg_pos={0 0}, upload=506004]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 29 16:41:34.649886370 (577h12m34.520969406s) [noise=true , address=45.91.116.10:4081 , piece_id=YBD6GEZPZ7RPLTMWHGI6CEJ3RSMEQAYJXWAQJK4A7QXMHDBNM6XA, seg_pos={0 0}, upload=506004]: Long tail cancellation due to optimal threshold Dec 29 16:41:34.649905858 (577h12m34.520988893s) [seg_pos={0 0}, upload=506004]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:41:34.649981913 (577h12m34.521064958s) [upload=506004]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:41:34.649989878 (577h12m34.521072923s) [upload=506004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:41:34.942904321 (577h12m34.813987356s) [upload=506004]: Done uploading segments... Dec 29 16:41:34.942923487 (577h12m34.814006522s) [upload=506004]: Upload finished. err: <nil> Duration: 30.003261194s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..X] duration: 5.001094558s trace: https://jaeger.dev.storj.io/trace/78df9d2d7ea2b017 FAIL (2025-12-29 16:41:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001117202s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.......X..] duration: 48.663448059s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:41:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 48.663461785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X......X.....X......X...XX.XXX....X...XXXXX.X..XX...XX..XXX] duration: 30.003393994s trace: https://jaeger.dev.storj.io/trace/4acb039a3efe80fe FAIL (2025-12-29 16:41:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:40:51.540140987 (577h11m51.411224032s) [upload=505994]: Starting upload Dec 29 16:40:51.540220157 (577h11m51.411303222s) [upload=505994]: Done starting upload Dec 29 16:40:51.540351207 (577h11m51.411434262s) [upload=505994]: Uploading segments... Dec 29 16:40:51.540530538 (577h11m51.411613593s) [upload=505994]: Got next segment. Inline: false Dec 29 16:40:51.540608237 (577h11m51.411691292s) [upload=505994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:40:51.540615010 (577h11m51.411698065s) [upload=505994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:40:51.788205028 (577h11m51.659288073s) [seg_pos={0 0}, upload=505994]: Begin upload segment... Dec 29 16:40:51.788251787 (577h11m51.659334832s) [seg_pos={0 0}, upload=505994]: Done begin upload segment. Dec 29 16:40:51.788278388 (577h11m51.659361423s) [noise=true , address=145.79.227.50:4375 , piece_id=CGZZD3PKSC75S3XUQTGD5AJGPTL573C4G2J2O42XEAVQHDNV2OWA, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.788307102 (577h11m51.659390147s) [noise=true , address=77.90.31.16:4735 , piece_id=DVGDVKQBFANGZSZZ7Q3AA3YB6HSRAFG6YPC2NOZ2HTMNG5BOKAEQ, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.788791174 (577h11m51.659874219s) [noise=true , address=145.79.227.51:4411 , piece_id=HFH37MRV7ERS4GLUEKAXJZKZCRJFXQZKCUUJDUTC5ZNTW77STIPQ, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.789381578 (577h11m51.660464623s) [noise=true , address=45.91.116.6:4974 , piece_id=XSBEQPAXDAZS54AO42QZHYX34Y74BK3CIKV3K4XARDR74GHHQEFQ, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.789688402 (577h11m51.660771447s) [noise=true , address=145.79.227.3:4149 , piece_id=Y6LNFTTP6AQTFGKNK3NH4CAB5JX4W6A3TAQYAIFX4XAQDEATFQ4A, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.790244742 (577h11m51.661327787s) [noise=true , address=45.91.116.10:4082 , piece_id=IK53IRDEF3KJZOXRIOI6U3XLVG4ZSJPZIAAXHVAF5PUKOE6YJYIQ, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.790359891 (577h11m51.661442926s) [noise=true , address=145.79.227.46:4602 , piece_id=HNZFBGCZX3AHCVYCPNFS4XYPJLFLMVXPCPLEDHEE3P6LKF4HX4QA, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.790880983 (577h11m51.661964018s) [noise=true , address=77.90.31.17:4770 , piece_id=NII4RD4E3C2TGAIOV66MM7RUZYCPW5L6EPS5MPAC7RXRXPZPCWPQ, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.791261468 (577h11m51.662344503s) [noise=true , address=145.79.227.22:4509 , piece_id=76D2Q3YBFTKYEXDJE3B7OR6XMYKCW2VRMNHRFNBKYXEGQFFXJ5SQ, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.791541771 (577h11m51.662624806s) [noise=true , address=145.79.227.26:4653 , piece_id=JZK7A6H6W5BESOCVYNDOLVRBGKCJGEQNHGHTS2MB4VQF2E7AW5CQ, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.791625141 (577h11m51.662708176s) [noise=true , address=145.79.227.39:4014 , piece_id=EZTNASXBVZR6VEV7LHCYPHTBA5P4YJXWF27FYFCRBE64PSFTO4NA, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.791877491 (577h11m51.662960526s) [noise=true , address=145.79.227.38:4977 , piece_id=SGH4M5OXH7QOX76T3DZNSI4GF6PG3XNJLHTUHWHHVVKDBG7JOTBQ, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.791977181 (577h11m51.663060216s) [noise=true , address=45.91.116.5:4297 , piece_id=ZQWODQF6AG5DUYEPU5MWI56DOAWM2SVECRLUTDEVW7NENAJPK3GA, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.792305486 (577h11m51.663388531s) [noise=true , address=145.79.227.48:4303 , piece_id=ZCM34T4CDOWKMHONVMCKW5VJQQCI5BOT4VQM4KPX6QQQDQAPVWXA, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.792626066 (577h11m51.663709101s) [noise=true , address=145.79.227.19:4079 , piece_id=MVQT6AVJDB3WWWTGSYFUVFEODQ3F6EQLN43ROFYWFNLJWWOT7OSA, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:51.793150114 (577h11m51.664233159s) [noise=true , address=77.90.31.11:4556 , piece_id=XW5GL4MT4GQ6ETISCHMOCHUMD3OR2SJ7YFWYA7EGYFPKSWVMR3VA, seg_pos={0 0}, upload=505994]: Uploading piece... Dec 29 16:40:53.479789216 (577h11m53.350872261s) [upload=505994]: Next returned nil segment Dec 29 16:40:53.479810085 (577h11m53.35089313s) [upload=505994]: Waiting for error group managing segments... Dec 29 16:40:53.765900504 (577h11m53.636983549s) [noise=true , address=145.79.227.3:4149 , piece_id=Y6LNFTTP6AQTFGKNK3NH4CAB5JX4W6A3TAQYAIFX4XAQDEATFQ4A, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:53.766107589 (577h11m53.637190634s) [noise=true , address=145.79.227.50:4375 , piece_id=CGZZD3PKSC75S3XUQTGD5AJGPTL573C4G2J2O42XEAVQHDNV2OWA, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:53.775981069 (577h11m53.647064114s) [noise=true , address=145.79.227.46:4602 , piece_id=HNZFBGCZX3AHCVYCPNFS4XYPJLFLMVXPCPLEDHEE3P6LKF4HX4QA, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:53.777517705 (577h11m53.64860075s) [noise=true , address=145.79.227.48:4303 , piece_id=ZCM34T4CDOWKMHONVMCKW5VJQQCI5BOT4VQM4KPX6QQQDQAPVWXA, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:53.781716701 (577h11m53.652799746s) [noise=true , address=145.79.227.26:4653 , piece_id=JZK7A6H6W5BESOCVYNDOLVRBGKCJGEQNHGHTS2MB4VQF2E7AW5CQ, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:53.794919280 (577h11m53.666002325s) [noise=true , address=145.79.227.38:4977 , piece_id=SGH4M5OXH7QOX76T3DZNSI4GF6PG3XNJLHTUHWHHVVKDBG7JOTBQ, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:53.795785219 (577h11m53.666868254s) [noise=true , address=145.79.227.19:4079 , piece_id=MVQT6AVJDB3WWWTGSYFUVFEODQ3F6EQLN43ROFYWFNLJWWOT7OSA, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:53.798256084 (577h11m53.669339119s) [noise=true , address=145.79.227.39:4014 , piece_id=EZTNASXBVZR6VEV7LHCYPHTBA5P4YJXWF27FYFCRBE64PSFTO4NA, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:53.804799033 (577h11m53.675882068s) [noise=true , address=145.79.227.22:4509 , piece_id=76D2Q3YBFTKYEXDJE3B7OR6XMYKCW2VRMNHRFNBKYXEGQFFXJ5SQ, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:53.809318829 (577h11m53.680401864s) [noise=true , address=145.79.227.51:4411 , piece_id=HFH37MRV7ERS4GLUEKAXJZKZCRJFXQZKCUUJDUTC5ZNTW77STIPQ, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:53.961887783 (577h11m53.832970818s) [noise=true , address=77.90.31.11:4556 , piece_id=XW5GL4MT4GQ6ETISCHMOCHUMD3OR2SJ7YFWYA7EGYFPKSWVMR3VA, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:53.971642778 (577h11m53.842725813s) [noise=true , address=77.90.31.16:4735 , piece_id=DVGDVKQBFANGZSZZ7Q3AA3YB6HSRAFG6YPC2NOZ2HTMNG5BOKAEQ, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:54.049308946 (577h11m53.920391982s) [noise=true , address=45.91.116.5:4297 , piece_id=ZQWODQF6AG5DUYEPU5MWI56DOAWM2SVECRLUTDEVW7NENAJPK3GA, seg_pos={0 0}, upload=505994]: Done uploading piece. err: <nil> Dec 29 16:40:54.049332521 (577h11m53.920415556s) [seg_pos={0 0}, upload=505994]: Segment reached optimal threshold of 13 pieces. Dec 29 16:40:54.049580603 (577h11m53.920663648s) [noise=true , address=45.91.116.10:4082 , piece_id=IK53IRDEF3KJZOXRIOI6U3XLVG4ZSJPZIAAXHVAF5PUKOE6YJYIQ, seg_pos={0 0}, upload=505994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 16:40:54.049590623 (577h11m53.920673688s) [noise=true , address=45.91.116.10:4082 , piece_id=IK53IRDEF3KJZOXRIOI6U3XLVG4ZSJPZIAAXHVAF5PUKOE6YJYIQ, seg_pos={0 0}, upload=505994]: Long tail cancellation due to optimal threshold Dec 29 16:40:54.049635929 (577h11m53.920718974s) [noise=true , address=77.90.31.17:4770 , piece_id=NII4RD4E3C2TGAIOV66MM7RUZYCPW5L6EPS5MPAC7RXRXPZPCWPQ, seg_pos={0 0}, upload=505994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 16:40:54.049646118 (577h11m53.920729153s) [noise=true , address=77.90.31.17:4770 , piece_id=NII4RD4E3C2TGAIOV66MM7RUZYCPW5L6EPS5MPAC7RXRXPZPCWPQ, seg_pos={0 0}, upload=505994]: Long tail cancellation due to optimal threshold Dec 29 16:40:54.049862379 (577h11m53.920945455s) [noise=true , address=45.91.116.6:4974 , piece_id=XSBEQPAXDAZS54AO42QZHYX34Y74BK3CIKV3K4XARDR74GHHQEFQ, seg_pos={0 0}, upload=505994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 16:40:54.049873221 (577h11m53.920956296s) [noise=true , address=45.91.116.6:4974 , piece_id=XSBEQPAXDAZS54AO42QZHYX34Y74BK3CIKV3K4XARDR74GHHQEFQ, seg_pos={0 0}, upload=505994]: Long tail cancellation due to optimal threshold Dec 29 16:40:54.049896334 (577h11m53.920979369s) [seg_pos={0 0}, upload=505994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:40:54.050005693 (577h11m53.921088758s) [upload=505994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:40:54.050014119 (577h11m53.921097154s) [upload=505994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:40:54.342378805 (577h11m54.21346184s) [upload=505994]: Done uploading segments... Dec 29 16:40:54.342396959 (577h11m54.213479994s) [upload=505994]: Upload finished. err: <nil> Duration: 30.003421015s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX..] duration: 5.001780242s trace: https://jaeger.dev.storj.io/trace/13cdfa7aa9c2c7ae FAIL (2025-12-29 16:41:13 UTC) orbiter warning: context deadline exceeded Duration: 5.001806031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.......X.] duration: 40.91109827s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 40.911114601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X......X.....X......X...XX.XXX....X...XXXXX.X..XX...XX..XX] duration: 30.004385658s trace: https://jaeger.dev.storj.io/trace/73d3adb2d1612489 FAIL (2025-12-29 16:40:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: BKA25SAV72UVJZNQKAYMEEN22DKLIBXRM4TIM3PQAUL4WAYP7QZA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: ZJEJHS4WIMH6F2CNOFDGPXXRCZGLN7Y6TZPOVB3MYGKMK35RPYWA) context deadline exceeded ======LOGS====== 16:39:47.705631784 (577h10m47.576714829s) [upload=505982]: Starting upload Dec 29 16:39:47.705699714 (577h10m47.576782759s) [upload=505982]: Done starting upload Dec 29 16:39:47.705954209 (577h10m47.577037254s) [upload=505982]: Uploading segments... Dec 29 16:39:47.706009163 (577h10m47.577092218s) [upload=505982]: Got next segment. Inline: false Dec 29 16:39:47.706039431 (577h10m47.577122466s) [upload=505982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:39:47.706044761 (577h10m47.577127796s) [upload=505982]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:39:47.954306930 (577h10m47.825389965s) [seg_pos={0 0}, upload=505982]: Begin upload segment... Dec 29 16:39:47.954351334 (577h10m47.825434369s) [seg_pos={0 0}, upload=505982]: Done begin upload segment. Dec 29 16:39:47.954378506 (577h10m47.825461541s) [noise=true , address=145.79.227.15:4874 , piece_id=KSCCGJUO3ZXISN572LGL7NHM3R7NUJPHQDNF5QXDU73UWMAPQGPQ, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.954392553 (577h10m47.825475588s) [noise=true , address=145.79.227.28:4726 , piece_id=E45NY2NCUQLPMW3NF3NET6APOGNZNDVHQ2OL6E5VIANYKZAFLCOA, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.954493285 (577h10m47.82557633s) [noise=true , address=145.79.227.30:4762 , piece_id=LV2FNCC6GQXPFEETMDX6NWRN2HE3KGQFY4WARGMCGCRPGVU3W34Q, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.954905920 (577h10m47.825988955s) [noise=true , address=145.79.227.13:4804 , piece_id=JDIN6HGNMUOJH376JVRNS3IJFGMJ6GCJ2XCXHGJFST4RYNVLMZCQ, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.955289140 (577h10m47.826372185s) [noise=true , address=145.79.227.51:4411 , piece_id=NKGXWYI23LOQBWSMZW4CSITDKUKORQ5WVWAQA4FE45PQCMVP6IOA, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.955635580 (577h10m47.826718615s) [noise=true , address=45.91.116.12:4226 , piece_id=BKA25SAV72UVJZNQKAYMEEN22DKLIBXRM4TIM3PQAUL4WAYP7QZA, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.955910023 (577h10m47.826993058s) [noise=true , address=45.91.116.9:4264 , piece_id=PZIWGLBNA2P3ZHDEKR4YSXQWWDNAY532YUIZ7L5JLB52ARD7J5VQ, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.956182551 (577h10m47.827265586s) [noise=true , address=145.79.227.47:4265 , piece_id=BKGMF4V7Y6S3IP6NYQSJGQJBWNBKCAJZDQYUAVIZT67NCBWY3NWQ, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.956448638 (577h10m47.827531683s) [noise=true , address=45.91.116.7:4009 , piece_id=ZJEJHS4WIMH6F2CNOFDGPXXRCZGLN7Y6TZPOVB3MYGKMK35RPYWA, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.956733791 (577h10m47.827816826s) [noise=true , address=77.90.31.19:4841 , piece_id=EHEGG7RIMSBCBAS23PPH3JYOTIPJUBQ26WGTTUCRXKJ7H6YOLISA, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.957076784 (577h10m47.828159829s) [noise=true , address=145.79.227.6:4257 , piece_id=ONRG2I6VOG7DTBHYXFOKNYJMJXMRZYD4ZGJEIHJYXXRNPBRTCW2A, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.957473991 (577h10m47.828557026s) [noise=true , address=145.79.227.37:4566 , piece_id=OWY6I4KDL2C724BJU2VVX77P2ZQ7QD4W576UZQNK6EC43ZZQOVRA, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.958520273 (577h10m47.829603308s) [noise=true , address=145.79.227.31:4799 , piece_id=5ROCPTTCQLVE6ARZ3MQBOXDKJKYP2ICTM6MYZAWE7ORVBEZ2E42Q, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.959274468 (577h10m47.830357503s) [noise=true , address=145.79.227.16:4910 , piece_id=HO7L5T4QISC2KZAKKEUKUQ6IUJN3ZEWDMOPEBZ7CVWPBIFR3JQJQ, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.960883883 (577h10m47.831966918s) [noise=true , address=145.79.227.17:4971 , piece_id=ODEJEFS4OZF4IXCRCGOMCNXCLJTZ4MW4XCB2VNCXTLLBRUGJPGXA, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:47.961853960 (577h10m47.832936995s) [noise=true , address=145.79.227.18:4007 , piece_id=5TPF7MYDSMLWDJCVS3P5KG4PW5XUQNSAM5WROAUJNV6XAKP357QQ, seg_pos={0 0}, upload=505982]: Uploading piece... Dec 29 16:39:49.662000651 (577h10m49.533083686s) [upload=505982]: Next returned nil segment Dec 29 16:39:49.662029587 (577h10m49.533112632s) [upload=505982]: Waiting for error group managing segments... Dec 29 16:39:49.946214287 (577h10m49.817297332s) [noise=true , address=145.79.227.51:4411 , piece_id=NKGXWYI23LOQBWSMZW4CSITDKUKORQ5WVWAQA4FE45PQCMVP6IOA, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:49.954321454 (577h10m49.825404489s) [noise=true , address=145.79.227.13:4804 , piece_id=JDIN6HGNMUOJH376JVRNS3IJFGMJ6GCJ2XCXHGJFST4RYNVLMZCQ, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:49.954708972 (577h10m49.825792007s) [noise=true , address=145.79.227.15:4874 , piece_id=KSCCGJUO3ZXISN572LGL7NHM3R7NUJPHQDNF5QXDU73UWMAPQGPQ, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:49.956179682 (577h10m49.827262717s) [noise=true , address=145.79.227.16:4910 , piece_id=HO7L5T4QISC2KZAKKEUKUQ6IUJN3ZEWDMOPEBZ7CVWPBIFR3JQJQ, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:49.961075505 (577h10m49.83215854s) [noise=true , address=145.79.227.31:4799 , piece_id=5ROCPTTCQLVE6ARZ3MQBOXDKJKYP2ICTM6MYZAWE7ORVBEZ2E42Q, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:49.964815656 (577h10m49.835898691s) [noise=true , address=145.79.227.47:4265 , piece_id=BKGMF4V7Y6S3IP6NYQSJGQJBWNBKCAJZDQYUAVIZT67NCBWY3NWQ, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:49.967731649 (577h10m49.838814694s) [noise=true , address=145.79.227.18:4007 , piece_id=5TPF7MYDSMLWDJCVS3P5KG4PW5XUQNSAM5WROAUJNV6XAKP357QQ, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:49.974555593 (577h10m49.845638628s) [noise=true , address=145.79.227.28:4726 , piece_id=E45NY2NCUQLPMW3NF3NET6APOGNZNDVHQ2OL6E5VIANYKZAFLCOA, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:49.981337877 (577h10m49.852420922s) [noise=true , address=145.79.227.37:4566 , piece_id=OWY6I4KDL2C724BJU2VVX77P2ZQ7QD4W576UZQNK6EC43ZZQOVRA, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:49.982019676 (577h10m49.853102711s) [noise=true , address=145.79.227.17:4971 , piece_id=ODEJEFS4OZF4IXCRCGOMCNXCLJTZ4MW4XCB2VNCXTLLBRUGJPGXA, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:50.264916754 (577h10m50.135999789s) [noise=true , address=45.91.116.12:4226 , piece_id=BKA25SAV72UVJZNQKAYMEEN22DKLIBXRM4TIM3PQAUL4WAYP7QZA, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:50.293610892 (577h10m50.164693937s) [noise=true , address=45.91.116.9:4264 , piece_id=PZIWGLBNA2P3ZHDEKR4YSXQWWDNAY532YUIZ7L5JLB52ARD7J5VQ, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:50.867507316 (577h10m50.738590351s) [noise=true , address=45.91.116.7:4009 , piece_id=ZJEJHS4WIMH6F2CNOFDGPXXRCZGLN7Y6TZPOVB3MYGKMK35RPYWA, seg_pos={0 0}, upload=505982]: Done uploading piece. err: <nil> Dec 29 16:39:50.867525301 (577h10m50.738608336s) [seg_pos={0 0}, upload=505982]: Segment reached optimal threshold of 13 pieces. Dec 29 16:39:50.867699272 (577h10m50.738782307s) [noise=true , address=145.79.227.30:4762 , piece_id=LV2FNCC6GQXPFEETMDX6NWRN2HE3KGQFY4WARGMCGCRPGVU3W34Q, seg_pos={0 0}, upload=505982]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:50.867709100 (577h10m50.738792135s) [noise=true , address=145.79.227.30:4762 , piece_id=LV2FNCC6GQXPFEETMDX6NWRN2HE3KGQFY4WARGMCGCRPGVU3W34Q, seg_pos={0 0}, upload=505982]: Long tail cancellation due to optimal threshold Dec 29 16:39:50.867811425 (577h10m50.73889446s) [noise=true , address=77.90.31.19:4841 , piece_id=EHEGG7RIMSBCBAS23PPH3JYOTIPJUBQ26WGTTUCRXKJ7H6YOLISA, seg_pos={0 0}, upload=505982]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:50.867816595 (577h10m50.73889964s) [noise=true , address=77.90.31.19:4841 , piece_id=EHEGG7RIMSBCBAS23PPH3JYOTIPJUBQ26WGTTUCRXKJ7H6YOLISA, seg_pos={0 0}, upload=505982]: Long tail cancellation due to optimal threshold Dec 29 16:39:50.867929470 (577h10m50.739012505s) [noise=true , address=145.79.227.6:4257 , piece_id=ONRG2I6VOG7DTBHYXFOKNYJMJXMRZYD4ZGJEIHJYXXRNPBRTCW2A, seg_pos={0 0}, upload=505982]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:55994->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:55994->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55994->145.79.227.6:4257: use of closed network connection Dec 29 16:39:50.867934550 (577h10m50.739017595s) [noise=true , address=145.79.227.6:4257 , piece_id=ONRG2I6VOG7DTBHYXFOKNYJMJXMRZYD4ZGJEIHJYXXRNPBRTCW2A, seg_pos={0 0}, upload=505982]: Long tail cancellation due to optimal threshold Dec 29 16:39:50.867950410 (577h10m50.739033455s) [seg_pos={0 0}, upload=505982]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:39:50.868014081 (577h10m50.739097126s) [upload=505982]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:50.868018349 (577h10m50.739101384s) [upload=505982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:51.169101126 (577h10m51.040184161s) [upload=505982]: Done uploading segments... Dec 29 16:39:51.169124671 (577h10m51.040207716s) [upload=505982]: Upload finished. err: <nil> Duration: 30.004409824s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [X.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX.] duration: 5.000859717s trace: https://jaeger.dev.storj.io/trace/13cdfa7aa9c2c7ae FAIL (2025-12-29 16:40:17 UTC) orbiter warning: context deadline exceeded Duration: 5.000885806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.......X] duration: 1m0.003424509s trace: https://jaeger.dev.storj.io/trace/72dea15b1cf83a3e FAIL (2025-12-29 16:40:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm, Piece ID: O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA) context deadline exceeded; piecestore: (Node ID: 12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, Piece ID: CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA) context deadline exceeded ======LOGS====== pload=505970]: Uploading piece... Dec 29 16:39:01.562157701 (577h10m1.433240746s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.562847736 (577h10m1.433930781s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.563881604 (577h10m1.43496465s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564374894 (577h10m1.435457939s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.564805573 (577h10m1.435888618s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565369107 (577h10m1.436452162s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.565704585 (577h10m1.43678763s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566266074 (577h10m1.437349109s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.566793198 (577h10m1.437876243s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568238711 (577h10m1.439321756s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.568609728 (577h10m1.439692773s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.569669916 (577h10m1.440752961s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.570646235 (577h10m1.44172928s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.571377468 (577h10m1.442460513s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572829763 (577h10m1.443912808s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.572864269 (577h10m1.443947314s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:01.575345894 (577h10m1.446428939s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Uploading piece... Dec 29 16:39:02.475505402 (577h10m2.346588467s) [noise=true , address=64.49.13.26:4945 , piece_id=MOFJUTEQDFS3H5EDOI6XN4F7T7OJ4RE4QY6PMXTLPY6ZPCAZJE2Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: ecclient: upload failed (node:12aPQ1AM7icmWJDhLQtS8pu3FCnun1peLjUdKZ7uBpAQ9yYrsYx, address:64.49.13.26:4945): protocol: storage node overloaded, request limit: 2000 Dec 29 16:39:02.887311432 (577h10m2.758394477s) [upload=505970]: Next returned nil segment Dec 29 16:39:02.887325559 (577h10m2.758408594s) [upload=505970]: Waiting for error group managing segments... Dec 29 16:39:03.239078038 (577h10m3.110161083s) [noise=true , address=64.49.13.154:4935 , piece_id=BBLIF5ICQ5YU2FSQN6Z4VEBFBZ6JRMSBLP6F5JYHWPXGO237CVXA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.258272617 (577h10m3.129355672s) [noise=true , address=64.49.13.139:4475 , piece_id=UTC2WVSERR2ISPM7GSPLABDV7P7KY4IFMOEGP377KQUEUHQYHY5Q, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.505913693 (577h10m3.376996738s) [noise=true , address=145.79.227.29:4964 , piece_id=XCVABUO6FFXHBGWCU7NHKNN3XZ75X7Z5EMC7AH6NJWCJV6YAYB4A, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.722993886 (577h10m3.594076921s) [noise=true , address=45.91.116.4:4948 , piece_id=RKQSHDBTHL4HS3QJ3ZVSWDALWE4T54A5RFMCN4KFOM43NAVOSIGA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.734287721 (577h10m3.605370766s) [noise=true , address=45.91.116.5:4307 , piece_id=MK34QPYVLEVQZBZO73ZORP55JMFYLSYRXU7PXHZIBP2VFYI34JRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.805982669 (577h10m3.677065714s) [noise=true , address=145.79.227.2:4130 , piece_id=RDQA4SKAPYKIW6AP7OLEX7K27MCEHHEG2NVIB476YCAGQK2MW7RQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.831111207 (577h10m3.702194252s) [noise=true , address=45.91.116.9:4283 , piece_id=ADTQTSQPOEW3MG57YUMAZCSDWVFILY5Q6GSGBXJ5ISUJ3CZHZ4VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:03.866146740 (577h10m3.737229785s) [noise=true , address=77.90.31.13:4639 , piece_id=CMCF6JUHNVWQD4IZIOJUOIW7RRVQJQLCA55U7LWQPEJCWXL4E3DQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.359720558 (577h10m4.230803593s) [noise=true , address=77.90.31.12:4607 , piece_id=O54HQWPCUYWMTTYBNGBLHIYTYYYPA5LNTES32MVWLVRKU7LR74VA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388423091 (577h10m4.259506136s) [noise=true , address=145.79.227.30:4783 , piece_id=KGWKJEBY723RBW2J6V4QV4JFW2SC5F3RANE7YWV7XS4AJQNA57BA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.388489267 (577h10m4.259572312s) [noise=true , address=145.79.227.44:4216 , piece_id=JGEYHE6LJZTWKYCQR2RVRC45XT5W2IRUMTVDWZU4OS24R6GOI2TQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.421923872 (577h10m4.293006907s) [noise=true , address=77.90.31.17:4777 , piece_id=7PH4UHSY6AQIOY2IKWLSPBQUMROYD2KOG6VFESWZXQWL7YPTSZMQ, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428192298 (577h10m4.299275333s) [noise=true , address=145.79.227.34:4931 , piece_id=LS4JJ4LFPQQDQBWNAUAQ3Z4T4GVYF7JUTB65QVYACPK3KJYBDSBA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428797110 (577h10m4.299880155s) [noise=true , address=145.79.227.34:4913 , piece_id=MYJIZ7EW7KSIH2YJBY3G4UZXLISL5GFKOML26UJOY2IAVZDDJ6SA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: <nil> Dec 29 16:39:04.428812840 (577h10m4.299895885s) [seg_pos={0 0}, upload=505970]: Segment reached optimal threshold of 14 pieces. Dec 29 16:39:04.429144160 (577h10m4.300227196s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34052->64.49.13.169:4709: use of closed network connection Dec 29 16:39:04.429168257 (577h10m4.300251292s) [noise=true , address=64.49.13.169:4709 , piece_id=D2FFMN5DYBLM3OESYS6TNUZPJW5IEYWQTWGVYHMOAFTZVIUV3XRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429564471 (577h10m4.300647516s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429588547 (577h10m4.300671592s) [noise=true , address=77.90.31.16:4762 , piece_id=6LC77G4YCJ3DBKQPZDIFT36I6LHZTZVQGGCZMIHULP5CQPAAHJQA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429949794 (577h10m4.301032839s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF Dec 29 16:39:04.429961938 (577h10m4.301044983s) [noise=true , address=145.79.227.33:4903 , piece_id=6VKBPOIYE3XEHWYYIGOP73Z2OXLH7626CGCO3O5SDWRQOIP2NQRA, seg_pos={0 0}, upload=505970]: Long tail cancellation due to optimal threshold Dec 29 16:39:04.429997265 (577h10m4.30108031s) [seg_pos={0 0}, upload=505970]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:39:04.430120640 (577h10m4.301203685s) [upload=505970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:39:04.430128374 (577h10m4.301211439s) [upload=505970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:39:04.726418535 (577h10m4.597501571s) [upload=505970]: Done uploading segments... Dec 29 16:39:04.726438423 (577h10m4.597521458s) [upload=505970]: Upload finished. err: <nil> Duration: 1m0.003434318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXXX] duration: 5.000765182s trace: https://jaeger.dev.storj.io/trace/13cdfa7aa9c2c7ae FAIL (2025-12-29 16:39:38 UTC) orbiter warning: context deadline exceeded Duration: 5.000787875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X......X.....X......X...XX.XXX....X...XXXXX.X..XX...XX..X] duration: 30.003641974s trace: https://jaeger.dev.storj.io/trace/47d6229457b8f09f FAIL (2025-12-29 16:39:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: VQJYQOIUADBNYZ52PQIB4TF7NYEEGX3RR7PLDYXWLNGECPDM5FZQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: E7P5URVGBQDHJZJRTCMJQDBEI2QIZ2IKF7DGQHRLAN5SKS4CX2CA) context deadline exceeded ======LOGS====== Dec 29 16:38:42.195517299 (577h9m42.066600334s) [upload=505966]: Starting upload Dec 29 16:38:42.195571723 (577h9m42.066654758s) [upload=505966]: Done starting upload Dec 29 16:38:42.195747558 (577h9m42.066830593s) [upload=505966]: Uploading segments... Dec 29 16:38:42.195786722 (577h9m42.066869767s) [upload=505966]: Got next segment. Inline: false Dec 29 16:38:42.195824103 (577h9m42.066907148s) [upload=505966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:38:42.195830195 (577h9m42.06691323s) [upload=505966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:38:42.438875901 (577h9m42.309958946s) [seg_pos={0 0}, upload=505966]: Begin upload segment... Dec 29 16:38:42.438947216 (577h9m42.310030262s) [seg_pos={0 0}, upload=505966]: Done begin upload segment. Dec 29 16:38:42.438965732 (577h9m42.310048777s) [noise=true , address=145.79.227.7:4293 , piece_id=33CCTZLAMR3IWHMB7VX5EVXFJZDCMD6DPXWXTT6SGZH7JWK7Y25Q, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.438990338 (577h9m42.310073373s) [noise=true , address=45.91.116.5:4298 , piece_id=C4IUPBZ3HBP3D34MKW524EFX5DCWJ64J4EVRNUHQ2MULE7G4QJFQ, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.439060001 (577h9m42.310143066s) [noise=true , address=77.90.31.21:4878 , piece_id=S4S5KTM2WQ6D64KPT7IE53SZPHTI4TAVJH53LBKUKTB73CB4VQNQ, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.439694049 (577h9m42.310777114s) [noise=true , address=45.91.116.14:4190 , piece_id=E7P5URVGBQDHJZJRTCMJQDBEI2QIZ2IKF7DGQHRLAN5SKS4CX2CA, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.440281918 (577h9m42.311364953s) [noise=true , address=77.90.31.18:4808 , piece_id=KZHNANTOYYMY53VSNZ7CZS3NEEWYY7NS25ERO2IPNR67PLCDGFTQ, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.440640301 (577h9m42.311723336s) [noise=true , address=145.79.227.4:4187 , piece_id=DY42NU2FKA5KIMTS7YBW7ZJTXOAYUET7372F23HGXXHIYEY7JZ7Q, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.441076541 (577h9m42.312159596s) [noise=true , address=145.79.227.17:4971 , piece_id=PXBXJCDE2SXXUZKMHIIR6ADMSZ6TQY5LTCULDCOF2EBFBTOLY6GA, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.441463257 (577h9m42.312546292s) [noise=true , address=145.79.227.38:4979 , piece_id=KX7PHOWE5NDEQUYPSFVRZ3IKYFVHLML6BBBUBFR36NJUHAQJ3CJQ, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.441944504 (577h9m42.313027549s) [noise=true , address=77.90.31.13:4626 , piece_id=AP565OBQU6KMOWGACWRAMB4KKAU2ORGKBVDNIZHY3MZVAILUUA6A, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.442487378 (577h9m42.313570423s) [noise=true , address=145.79.227.52:4447 , piece_id=GTCAFIZWYVH3QDYCYJ2AYWZQO5O3SPVRROXHHVS6BYYG6DGJZVHQ, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.445935734 (577h9m42.317018769s) [noise=true , address=145.79.227.51:4411 , piece_id=XNMABQRE53TNFNPHMMV5ZYMAITSZV7H24QRZQP4BPMQJD5GPLAOA, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.448711188 (577h9m42.319794233s) [noise=true , address=45.91.116.7:4012 , piece_id=RZNHXPQAXPVZLEQEQOBAV4OJJJQ565BLM7VJ6SEFUTQT6YIM72FQ, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.457061088 (577h9m42.328144133s) [noise=true , address=45.91.116.13:4156 , piece_id=VQJYQOIUADBNYZ52PQIB4TF7NYEEGX3RR7PLDYXWLNGECPDM5FZQ, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.457481318 (577h9m42.328564353s) [noise=true , address=145.79.227.48:4301 , piece_id=WLA5LIPXCGFLPPAQGLGIGEU4RGH5VNZLGR4O5VAPYKRF6IKXZAIQ, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.461141247 (577h9m42.332224292s) [noise=true , address=145.79.227.50:4373 , piece_id=JVVUPDWS5YIEN2MDAQFENAVQTFDOMZAX5KFBJNKNHIWTF7IITBOQ, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:42.463351005 (577h9m42.334434051s) [noise=true , address=145.79.227.37:4567 , piece_id=SWORMG7JKCQSMMG6ADLARZ2FF3ZSTRXKICDW7DS5HKPBVG6KG7KQ, seg_pos={0 0}, upload=505966]: Uploading piece... Dec 29 16:38:44.156587037 (577h9m44.027670072s) [upload=505966]: Next returned nil segment Dec 29 16:38:44.156599972 (577h9m44.027683007s) [upload=505966]: Waiting for error group managing segments... Dec 29 16:38:44.435631759 (577h9m44.306714794s) [noise=true , address=145.79.227.7:4293 , piece_id=33CCTZLAMR3IWHMB7VX5EVXFJZDCMD6DPXWXTT6SGZH7JWK7Y25Q, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.443345518 (577h9m44.314428553s) [noise=true , address=145.79.227.50:4373 , piece_id=JVVUPDWS5YIEN2MDAQFENAVQTFDOMZAX5KFBJNKNHIWTF7IITBOQ, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.444002740 (577h9m44.315085775s) [noise=true , address=145.79.227.37:4567 , piece_id=SWORMG7JKCQSMMG6ADLARZ2FF3ZSTRXKICDW7DS5HKPBVG6KG7KQ, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.451245170 (577h9m44.322328216s) [noise=true , address=145.79.227.17:4971 , piece_id=PXBXJCDE2SXXUZKMHIIR6ADMSZ6TQY5LTCULDCOF2EBFBTOLY6GA, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.461073986 (577h9m44.332157021s) [noise=true , address=145.79.227.52:4447 , piece_id=GTCAFIZWYVH3QDYCYJ2AYWZQO5O3SPVRROXHHVS6BYYG6DGJZVHQ, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.461745584 (577h9m44.332828619s) [noise=true , address=145.79.227.48:4301 , piece_id=WLA5LIPXCGFLPPAQGLGIGEU4RGH5VNZLGR4O5VAPYKRF6IKXZAIQ, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.464678769 (577h9m44.335761815s) [noise=true , address=145.79.227.38:4979 , piece_id=KX7PHOWE5NDEQUYPSFVRZ3IKYFVHLML6BBBUBFR36NJUHAQJ3CJQ, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.469045164 (577h9m44.340128209s) [noise=true , address=145.79.227.51:4411 , piece_id=XNMABQRE53TNFNPHMMV5ZYMAITSZV7H24QRZQP4BPMQJD5GPLAOA, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.534731731 (577h9m44.405814766s) [noise=true , address=145.79.227.4:4187 , piece_id=DY42NU2FKA5KIMTS7YBW7ZJTXOAYUET7372F23HGXXHIYEY7JZ7Q, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.642075319 (577h9m44.513158354s) [noise=true , address=77.90.31.18:4808 , piece_id=KZHNANTOYYMY53VSNZ7CZS3NEEWYY7NS25ERO2IPNR67PLCDGFTQ, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.793540221 (577h9m44.664623266s) [noise=true , address=45.91.116.5:4298 , piece_id=C4IUPBZ3HBP3D34MKW524EFX5DCWJ64J4EVRNUHQ2MULE7G4QJFQ, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.818669090 (577h9m44.689752135s) [noise=true , address=45.91.116.14:4190 , piece_id=E7P5URVGBQDHJZJRTCMJQDBEI2QIZ2IKF7DGQHRLAN5SKS4CX2CA, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.892935785 (577h9m44.76401882s) [noise=true , address=45.91.116.13:4156 , piece_id=VQJYQOIUADBNYZ52PQIB4TF7NYEEGX3RR7PLDYXWLNGECPDM5FZQ, seg_pos={0 0}, upload=505966]: Done uploading piece. err: <nil> Dec 29 16:38:44.892999106 (577h9m44.764082141s) [seg_pos={0 0}, upload=505966]: Segment reached optimal threshold of 13 pieces. Dec 29 16:38:44.893237210 (577h9m44.764320255s) [noise=true , address=45.91.116.7:4012 , piece_id=RZNHXPQAXPVZLEQEQOBAV4OJJJQ565BLM7VJ6SEFUTQT6YIM72FQ, seg_pos={0 0}, upload=505966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 29 16:38:44.893246036 (577h9m44.764329081s) [noise=true , address=45.91.116.7:4012 , piece_id=RZNHXPQAXPVZLEQEQOBAV4OJJJQ565BLM7VJ6SEFUTQT6YIM72FQ, seg_pos={0 0}, upload=505966]: Long tail cancellation due to optimal threshold Dec 29 16:38:44.893395931 (577h9m44.764478976s) [noise=true , address=77.90.31.13:4626 , piece_id=AP565OBQU6KMOWGACWRAMB4KKAU2ORGKBVDNIZHY3MZVAILUUA6A, seg_pos={0 0}, upload=505966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 29 16:38:44.893417022 (577h9m44.764500057s) [noise=true , address=77.90.31.13:4626 , piece_id=AP565OBQU6KMOWGACWRAMB4KKAU2ORGKBVDNIZHY3MZVAILUUA6A, seg_pos={0 0}, upload=505966]: Long tail cancellation due to optimal threshold Dec 29 16:38:44.893577758 (577h9m44.764660803s) [noise=true , address=77.90.31.21:4878 , piece_id=S4S5KTM2WQ6D64KPT7IE53SZPHTI4TAVJH53LBKUKTB73CB4VQNQ, seg_pos={0 0}, upload=505966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:43636->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:43636->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43636->77.90.31.21:4878: use of closed network connection Dec 29 16:38:44.893587006 (577h9m44.764670041s) [noise=true , address=77.90.31.21:4878 , piece_id=S4S5KTM2WQ6D64KPT7IE53SZPHTI4TAVJH53LBKUKTB73CB4VQNQ, seg_pos={0 0}, upload=505966]: Long tail cancellation due to optimal threshold Dec 29 16:38:44.893610561 (577h9m44.764693606s) [seg_pos={0 0}, upload=505966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:38:44.893691505 (577h9m44.76477455s) [upload=505966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:38:44.893697336 (577h9m44.764780371s) [upload=505966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:38:45.191206186 (577h9m45.062289221s) [upload=505966]: Done uploading segments... Dec 29 16:38:45.191224822 (577h9m45.062307857s) [upload=505966]: Upload finished. err: <nil> Duration: 30.003668535s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XXX] duration: 5.001731251s trace: https://jaeger.dev.storj.io/trace/737f154b5b962593 FAIL (2025-12-29 16:38:36 UTC) orbiter warning: context deadline exceeded Duration: 5.001753524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 41.570958037s trace: https://jaeger.dev.storj.io/trace/25eb57d21bedad4d FAIL (2025-12-29 16:38:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, Piece ID: KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA) context deadline exceeded; piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ) context deadline exceeded; piecestore: (Node ID: 12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, Piece ID: MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA) context deadline exceeded ======LOGS====== 530948251s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660301397 (577h2m12.531384432s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660322538 (577h2m12.531405583s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660796670 (577h2m12.531879705s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661151005 (577h2m12.53223404s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661355344 (577h2m12.532438389s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661875545 (577h2m12.53295859s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662039136 (577h2m12.533122202s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662350129 (577h2m12.533433174s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662716055 (577h2m12.5337991s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662805526 (577h2m12.533888571s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.663913205 (577h2m12.53499624s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.664659345 (577h2m12.535742411s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.666957802 (577h2m12.538040847s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.668348230 (577h2m12.539431305s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:14.044161823 (577h2m13.915244868s) [upload=505855]: Next returned nil segment Dec 29 16:31:14.044175078 (577h2m13.915258113s) [upload=505855]: Waiting for error group managing segments... Dec 29 16:31:14.378564616 (577h2m14.249647661s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.513685195 (577h2m14.384768231s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.765759753 (577h2m14.636842788s) [noise=true , address=45.91.116.5:4316 , piece_id=BVVWSLENSRBMD5OBFJHPOBKDYHE55LLKPQ3MNLD4DCBPAKPUTGTA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.796869410 (577h2m14.667952445s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.805110142 (577h2m14.676193187s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.809656709 (577h2m14.680739744s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.828088656 (577h2m14.699171701s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.916522628 (577h2m14.787605663s) [noise=true , address=77.90.31.17:4795 , piece_id=QXXJ6VOBDRVCVRRMLSBQO7MB2DS5PUMDE7KPGOIJ77AJOJPSU7XQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.387028071 (577h2m15.258111106s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.424600454 (577h2m15.295683489s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.431096903 (577h2m15.302179939s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.454782875 (577h2m15.32586591s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.845291312 (577h2m15.716374357s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302138686 (577h2m16.173221731s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302163113 (577h2m16.173246148s) [seg_pos={0 0}, upload=505855]: Segment reached optimal threshold of 14 pieces. Dec 29 16:31:16.302359598 (577h2m16.173442663s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XmqDfLfXDjJ6Bo9PAYiVgshEBPnfQ1D9yC6CcvXpPRCWq4Wrs): protocol: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection Dec 29 16:31:16.302370097 (577h2m16.173453152s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302392490 (577h2m16.173475535s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection Dec 29 16:31:16.302401277 (577h2m16.173484322s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302574897 (577h2m16.173657952s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302582952 (577h2m16.173666018s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302723921 (577h2m16.173806956s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302731816 (577h2m16.173814851s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302750131 (577h2m16.173833176s) [seg_pos={0 0}, upload=505855]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:31:16.302842045 (577h2m16.17392509s) [upload=505855]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:31:16.302848428 (577h2m16.173931473s) [upload=505855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:31:16.589365678 (577h2m16.460448733s) [upload=505855]: Done uploading segments... Dec 29 16:31:16.589388892 (577h2m16.460471938s) [upload=505855]: Upload finished. err: <nil> Duration: 41.57098111s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.XX] duration: 5.001906085s trace: https://jaeger.dev.storj.io/trace/16dd7dbcaaf69617 FAIL (2025-12-29 16:38:00 UTC) orbiter warning: context deadline exceeded Duration: 5.001929289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X......X.....X......X...XX.XXX....X...XXXXX.X..XX...XX..] duration: 6.179277643s trace: https://jaeger.dev.storj.io/trace/2b8bccfebf290739 FAIL (2025-12-29 16:37:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:36:35.608651446 (577h7m35.479734491s) [upload=505935]: Starting upload Dec 29 16:36:35.608732149 (577h7m35.479815184s) [upload=505935]: Done starting upload Dec 29 16:36:35.608844242 (577h7m35.479927277s) [upload=505935]: Uploading segments... Dec 29 16:36:35.608934424 (577h7m35.480017459s) [upload=505935]: Got next segment. Inline: false Dec 29 16:36:35.609020818 (577h7m35.480103853s) [upload=505935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:36:35.609026500 (577h7m35.480109535s) [upload=505935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:36:35.854751968 (577h7m35.725835003s) [seg_pos={0 0}, upload=505935]: Begin upload segment... Dec 29 16:36:35.854793076 (577h7m35.725876111s) [seg_pos={0 0}, upload=505935]: Done begin upload segment. Dec 29 16:36:35.854818184 (577h7m35.725901229s) [noise=true , address=145.79.227.13:4803 , piece_id=OS57GYKLQ563PPPGNEM47UPYEXUIO3GLTV2HH2PX4JJW6SJ33ARQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.854831449 (577h7m35.725914484s) [noise=true , address=45.91.116.13:4154 , piece_id=XX7NAO3AGJQZRAHBUBDJLJ7H3GD634FRKBNPWLTHQRTDI6OS3MNQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.855216492 (577h7m35.726299527s) [noise=true , address=145.79.227.51:4411 , piece_id=HNFX255R67I5MJ54O5PEHT4QFQ7KOIYURPT47EOFVA7SS2PCKQHA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.855502437 (577h7m35.726585472s) [noise=true , address=145.79.227.45:4231 , piece_id=T2UFIYKIHAMH36PH2QYTJNJ6H5VTZWHBJQRVLHIVTWSI4ZY4VHIA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.855778292 (577h7m35.726861327s) [noise=true , address=145.79.227.6:4258 , piece_id=YRA6H2OGN6T67OZSAKY4UXPU3KSL3BPZRBRJL24GZVSAXUPKLDEA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856054969 (577h7m35.727138004s) [noise=true , address=45.91.116.6:4976 , piece_id=BP75OGDJNQ5QKWJXR5PAIFUF5SRS6RFNOX2AWETL72O2JLUQBM7A, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856307390 (577h7m35.727390425s) [noise=true , address=145.79.227.26:4653 , piece_id=RFBCUXNRC4ME5255WBK24JS7POEWAYZ4GMIXGNREOIO2A5UYYBEQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856554701 (577h7m35.727637746s) [noise=true , address=145.79.227.42:4123 , piece_id=6554I3WSWJ6USI4XH5KTXVATOAL2DS623MZD5TZRMWSJMY6BU47A, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856856014 (577h7m35.727939049s) [noise=true , address=77.90.31.16:4735 , piece_id=JQLZHLS2ATKE6ZF373M27CBACXR7SMXAHLGEMT7V4OHACAD5GY3Q, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.858488723 (577h7m35.729571768s) [noise=true , address=145.79.227.34:4905 , piece_id=IS52U3DJYFBPFVQTTGT5H6TR4NG5XKSJGW2DI3OBZJTLDZ5EL2GA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.858774286 (577h7m35.729857332s) [noise=true , address=145.79.227.9:4366 , piece_id=LD55E6A643OUSAP4GVRMUTUS4ZENGJQUX5ZSP5FK45AHADAPPOMA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.859827964 (577h7m35.730911009s) [noise=true , address=145.79.227.12:4765 , piece_id=2YVZUMYBBBHDNX3IUX4HHLRUPFDZJWMF5H44L3N3RR4YA4L3PZQA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.861024642 (577h7m35.732107777s) [noise=true , address=145.79.227.41:4085 , piece_id=GPCHKPBO7LZZHQ7EXRX5U4T6QWIND7YEAQAS6NQFIHTPEYNWAPNA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.862361888 (577h7m35.733444923s) [noise=true , address=45.91.116.4:4939 , piece_id=SQZSKGN6732LE3WMVHO6HVJ6VLK5VGUOGOBHIMVVHFJUW5FRNQNA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.863708903 (577h7m35.734791948s) [noise=true , address=145.79.227.50:4375 , piece_id=CMDNIPTQEFYQWBG3JLRXGML6FDTWCMU4GGSUVLDX7ZNQPZDSS7XA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.865083391 (577h7m35.736166426s) [noise=true , address=45.91.116.10:4084 , piece_id=XOMVJQE76AU3BNS5AVWIXUG6GHGAYVQT2VWAQ2N2MMI2PY6GISWQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:37.555310093 (577h7m37.426393138s) [upload=505935]: Next returned nil segment Dec 29 16:36:37.555330322 (577h7m37.426413357s) [upload=505935]: Waiting for error group managing segments... Dec 29 16:36:37.836205198 (577h7m37.707288233s) [noise=true , address=145.79.227.45:4231 , piece_id=T2UFIYKIHAMH36PH2QYTJNJ6H5VTZWHBJQRVLHIVTWSI4ZY4VHIA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.844111122 (577h7m37.715194157s) [noise=true , address=145.79.227.50:4375 , piece_id=CMDNIPTQEFYQWBG3JLRXGML6FDTWCMU4GGSUVLDX7ZNQPZDSS7XA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.844526874 (577h7m37.715609919s) [noise=true , address=145.79.227.26:4653 , piece_id=RFBCUXNRC4ME5255WBK24JS7POEWAYZ4GMIXGNREOIO2A5UYYBEQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.849170607 (577h7m37.720253652s) [noise=true , address=145.79.227.51:4411 , piece_id=HNFX255R67I5MJ54O5PEHT4QFQ7KOIYURPT47EOFVA7SS2PCKQHA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.850101289 (577h7m37.721184324s) [noise=true , address=145.79.227.12:4765 , piece_id=2YVZUMYBBBHDNX3IUX4HHLRUPFDZJWMF5H44L3N3RR4YA4L3PZQA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.858234125 (577h7m37.72931716s) [noise=true , address=145.79.227.42:4123 , piece_id=6554I3WSWJ6USI4XH5KTXVATOAL2DS623MZD5TZRMWSJMY6BU47A, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.858756961 (577h7m37.729840006s) [noise=true , address=145.79.227.13:4803 , piece_id=OS57GYKLQ563PPPGNEM47UPYEXUIO3GLTV2HH2PX4JJW6SJ33ARQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.866925414 (577h7m37.738008449s) [noise=true , address=145.79.227.9:4366 , piece_id=LD55E6A643OUSAP4GVRMUTUS4ZENGJQUX5ZSP5FK45AHADAPPOMA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.894357890 (577h7m37.765440925s) [noise=true , address=145.79.227.41:4085 , piece_id=GPCHKPBO7LZZHQ7EXRX5U4T6QWIND7YEAQAS6NQFIHTPEYNWAPNA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.115137568 (577h7m37.986220613s) [noise=true , address=45.91.116.13:4154 , piece_id=XX7NAO3AGJQZRAHBUBDJLJ7H3GD634FRKBNPWLTHQRTDI6OS3MNQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.246381235 (577h7m38.11746427s) [noise=true , address=45.91.116.6:4976 , piece_id=BP75OGDJNQ5QKWJXR5PAIFUF5SRS6RFNOX2AWETL72O2JLUQBM7A, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.427030097 (577h7m38.298113132s) [noise=true , address=45.91.116.4:4939 , piece_id=SQZSKGN6732LE3WMVHO6HVJ6VLK5VGUOGOBHIMVVHFJUW5FRNQNA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.810709200 (577h7m38.681792235s) [noise=true , address=77.90.31.16:4735 , piece_id=JQLZHLS2ATKE6ZF373M27CBACXR7SMXAHLGEMT7V4OHACAD5GY3Q, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.810862221 (577h7m38.681945266s) [seg_pos={0 0}, upload=505935]: Segment reached optimal threshold of 13 pieces. Dec 29 16:36:38.811163515 (577h7m38.68224658s) [noise=true , address=145.79.227.34:4905 , piece_id=IS52U3DJYFBPFVQTTGT5H6TR4NG5XKSJGW2DI3OBZJTLDZ5EL2GA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 16:36:38.811176659 (577h7m38.682259705s) [noise=true , address=145.79.227.34:4905 , piece_id=IS52U3DJYFBPFVQTTGT5H6TR4NG5XKSJGW2DI3OBZJTLDZ5EL2GA, seg_pos={0 0}, upload=505935]: Long tail cancellation due to optimal threshold Dec 29 16:36:38.811206516 (577h7m38.682289551s) [noise=true , address=145.79.227.6:4258 , piece_id=YRA6H2OGN6T67OZSAKY4UXPU3KSL3BPZRBRJL24GZVSAXUPKLDEA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:59834->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:59834->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59834->145.79.227.6:4258: use of closed network connection Dec 29 16:36:38.811239589 (577h7m38.682322625s) [noise=true , address=145.79.227.6:4258 , piece_id=YRA6H2OGN6T67OZSAKY4UXPU3KSL3BPZRBRJL24GZVSAXUPKLDEA, seg_pos={0 0}, upload=505935]: Long tail cancellation due to optimal threshold Dec 29 16:36:38.811510666 (577h7m38.682593711s) [noise=true , address=45.91.116.10:4084 , piece_id=XOMVJQE76AU3BNS5AVWIXUG6GHGAYVQT2VWAQ2N2MMI2PY6GISWQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 16:36:38.811520815 (577h7m38.68260386s) [noise=true , address=45.91.116.10:4084 , piece_id=XOMVJQE76AU3BNS5AVWIXUG6GHGAYVQT2VWAQ2N2MMI2PY6GISWQ, seg_pos={0 0}, upload=505935]: Long tail cancellation due to optimal threshold Dec 29 16:36:38.811545071 (577h7m38.682628116s) [seg_pos={0 0}, upload=505935]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:36:38.811713773 (577h7m38.682796858s) [upload=505935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:36:38.811721347 (577h7m38.682804412s) [upload=505935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:36:39.112152102 (577h7m38.983235137s) [upload=505935]: Done uploading segments... Dec 29 16:36:39.112171789 (577h7m38.983254834s) [upload=505935]: Upload finished. err: <nil> Duration: 6.179309864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 49.572403478s trace: https://jaeger.dev.storj.io/trace/25eb57d21bedad4d FAIL (2025-12-29 16:37:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, Piece ID: KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA) context deadline exceeded; piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ) context deadline exceeded; piecestore: (Node ID: 12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, Piece ID: MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA) context deadline exceeded ======LOGS====== 530948251s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660301397 (577h2m12.531384432s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660322538 (577h2m12.531405583s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660796670 (577h2m12.531879705s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661151005 (577h2m12.53223404s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661355344 (577h2m12.532438389s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661875545 (577h2m12.53295859s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662039136 (577h2m12.533122202s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662350129 (577h2m12.533433174s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662716055 (577h2m12.5337991s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662805526 (577h2m12.533888571s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.663913205 (577h2m12.53499624s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.664659345 (577h2m12.535742411s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.666957802 (577h2m12.538040847s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.668348230 (577h2m12.539431305s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:14.044161823 (577h2m13.915244868s) [upload=505855]: Next returned nil segment Dec 29 16:31:14.044175078 (577h2m13.915258113s) [upload=505855]: Waiting for error group managing segments... Dec 29 16:31:14.378564616 (577h2m14.249647661s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.513685195 (577h2m14.384768231s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.765759753 (577h2m14.636842788s) [noise=true , address=45.91.116.5:4316 , piece_id=BVVWSLENSRBMD5OBFJHPOBKDYHE55LLKPQ3MNLD4DCBPAKPUTGTA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.796869410 (577h2m14.667952445s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.805110142 (577h2m14.676193187s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.809656709 (577h2m14.680739744s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.828088656 (577h2m14.699171701s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.916522628 (577h2m14.787605663s) [noise=true , address=77.90.31.17:4795 , piece_id=QXXJ6VOBDRVCVRRMLSBQO7MB2DS5PUMDE7KPGOIJ77AJOJPSU7XQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.387028071 (577h2m15.258111106s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.424600454 (577h2m15.295683489s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.431096903 (577h2m15.302179939s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.454782875 (577h2m15.32586591s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.845291312 (577h2m15.716374357s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302138686 (577h2m16.173221731s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302163113 (577h2m16.173246148s) [seg_pos={0 0}, upload=505855]: Segment reached optimal threshold of 14 pieces. Dec 29 16:31:16.302359598 (577h2m16.173442663s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XmqDfLfXDjJ6Bo9PAYiVgshEBPnfQ1D9yC6CcvXpPRCWq4Wrs): protocol: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection Dec 29 16:31:16.302370097 (577h2m16.173453152s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302392490 (577h2m16.173475535s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection Dec 29 16:31:16.302401277 (577h2m16.173484322s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302574897 (577h2m16.173657952s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302582952 (577h2m16.173666018s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302723921 (577h2m16.173806956s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302731816 (577h2m16.173814851s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302750131 (577h2m16.173833176s) [seg_pos={0 0}, upload=505855]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:31:16.302842045 (577h2m16.17392509s) [upload=505855]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:31:16.302848428 (577h2m16.173931473s) [upload=505855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:31:16.589365678 (577h2m16.460448733s) [upload=505855]: Done uploading segments... Dec 29 16:31:16.589388892 (577h2m16.460471938s) [upload=505855]: Upload finished. err: <nil> Duration: 49.57243117s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.X] duration: 5.00108087s trace: https://jaeger.dev.storj.io/trace/6a99a14e9827a29a FAIL (2025-12-29 16:37:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001104865s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX.] duration: 5.000913993s trace: https://jaeger.dev.storj.io/trace/56715990b227e1e6 FAIL (2025-12-29 16:37:18 UTC) orbiter warning: context deadline exceeded Duration: 5.000938922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.X......X.....X......X...XX.XXX....X...XXXXX.X..XX...XX.] duration: 5.968935498s trace: https://jaeger.dev.storj.io/trace/2b8bccfebf290739 FAIL (2025-12-29 16:37:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:36:35.608651446 (577h7m35.479734491s) [upload=505935]: Starting upload Dec 29 16:36:35.608732149 (577h7m35.479815184s) [upload=505935]: Done starting upload Dec 29 16:36:35.608844242 (577h7m35.479927277s) [upload=505935]: Uploading segments... Dec 29 16:36:35.608934424 (577h7m35.480017459s) [upload=505935]: Got next segment. Inline: false Dec 29 16:36:35.609020818 (577h7m35.480103853s) [upload=505935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:36:35.609026500 (577h7m35.480109535s) [upload=505935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:36:35.854751968 (577h7m35.725835003s) [seg_pos={0 0}, upload=505935]: Begin upload segment... Dec 29 16:36:35.854793076 (577h7m35.725876111s) [seg_pos={0 0}, upload=505935]: Done begin upload segment. Dec 29 16:36:35.854818184 (577h7m35.725901229s) [noise=true , address=145.79.227.13:4803 , piece_id=OS57GYKLQ563PPPGNEM47UPYEXUIO3GLTV2HH2PX4JJW6SJ33ARQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.854831449 (577h7m35.725914484s) [noise=true , address=45.91.116.13:4154 , piece_id=XX7NAO3AGJQZRAHBUBDJLJ7H3GD634FRKBNPWLTHQRTDI6OS3MNQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.855216492 (577h7m35.726299527s) [noise=true , address=145.79.227.51:4411 , piece_id=HNFX255R67I5MJ54O5PEHT4QFQ7KOIYURPT47EOFVA7SS2PCKQHA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.855502437 (577h7m35.726585472s) [noise=true , address=145.79.227.45:4231 , piece_id=T2UFIYKIHAMH36PH2QYTJNJ6H5VTZWHBJQRVLHIVTWSI4ZY4VHIA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.855778292 (577h7m35.726861327s) [noise=true , address=145.79.227.6:4258 , piece_id=YRA6H2OGN6T67OZSAKY4UXPU3KSL3BPZRBRJL24GZVSAXUPKLDEA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856054969 (577h7m35.727138004s) [noise=true , address=45.91.116.6:4976 , piece_id=BP75OGDJNQ5QKWJXR5PAIFUF5SRS6RFNOX2AWETL72O2JLUQBM7A, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856307390 (577h7m35.727390425s) [noise=true , address=145.79.227.26:4653 , piece_id=RFBCUXNRC4ME5255WBK24JS7POEWAYZ4GMIXGNREOIO2A5UYYBEQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856554701 (577h7m35.727637746s) [noise=true , address=145.79.227.42:4123 , piece_id=6554I3WSWJ6USI4XH5KTXVATOAL2DS623MZD5TZRMWSJMY6BU47A, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856856014 (577h7m35.727939049s) [noise=true , address=77.90.31.16:4735 , piece_id=JQLZHLS2ATKE6ZF373M27CBACXR7SMXAHLGEMT7V4OHACAD5GY3Q, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.858488723 (577h7m35.729571768s) [noise=true , address=145.79.227.34:4905 , piece_id=IS52U3DJYFBPFVQTTGT5H6TR4NG5XKSJGW2DI3OBZJTLDZ5EL2GA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.858774286 (577h7m35.729857332s) [noise=true , address=145.79.227.9:4366 , piece_id=LD55E6A643OUSAP4GVRMUTUS4ZENGJQUX5ZSP5FK45AHADAPPOMA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.859827964 (577h7m35.730911009s) [noise=true , address=145.79.227.12:4765 , piece_id=2YVZUMYBBBHDNX3IUX4HHLRUPFDZJWMF5H44L3N3RR4YA4L3PZQA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.861024642 (577h7m35.732107777s) [noise=true , address=145.79.227.41:4085 , piece_id=GPCHKPBO7LZZHQ7EXRX5U4T6QWIND7YEAQAS6NQFIHTPEYNWAPNA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.862361888 (577h7m35.733444923s) [noise=true , address=45.91.116.4:4939 , piece_id=SQZSKGN6732LE3WMVHO6HVJ6VLK5VGUOGOBHIMVVHFJUW5FRNQNA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.863708903 (577h7m35.734791948s) [noise=true , address=145.79.227.50:4375 , piece_id=CMDNIPTQEFYQWBG3JLRXGML6FDTWCMU4GGSUVLDX7ZNQPZDSS7XA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.865083391 (577h7m35.736166426s) [noise=true , address=45.91.116.10:4084 , piece_id=XOMVJQE76AU3BNS5AVWIXUG6GHGAYVQT2VWAQ2N2MMI2PY6GISWQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:37.555310093 (577h7m37.426393138s) [upload=505935]: Next returned nil segment Dec 29 16:36:37.555330322 (577h7m37.426413357s) [upload=505935]: Waiting for error group managing segments... Dec 29 16:36:37.836205198 (577h7m37.707288233s) [noise=true , address=145.79.227.45:4231 , piece_id=T2UFIYKIHAMH36PH2QYTJNJ6H5VTZWHBJQRVLHIVTWSI4ZY4VHIA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.844111122 (577h7m37.715194157s) [noise=true , address=145.79.227.50:4375 , piece_id=CMDNIPTQEFYQWBG3JLRXGML6FDTWCMU4GGSUVLDX7ZNQPZDSS7XA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.844526874 (577h7m37.715609919s) [noise=true , address=145.79.227.26:4653 , piece_id=RFBCUXNRC4ME5255WBK24JS7POEWAYZ4GMIXGNREOIO2A5UYYBEQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.849170607 (577h7m37.720253652s) [noise=true , address=145.79.227.51:4411 , piece_id=HNFX255R67I5MJ54O5PEHT4QFQ7KOIYURPT47EOFVA7SS2PCKQHA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.850101289 (577h7m37.721184324s) [noise=true , address=145.79.227.12:4765 , piece_id=2YVZUMYBBBHDNX3IUX4HHLRUPFDZJWMF5H44L3N3RR4YA4L3PZQA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.858234125 (577h7m37.72931716s) [noise=true , address=145.79.227.42:4123 , piece_id=6554I3WSWJ6USI4XH5KTXVATOAL2DS623MZD5TZRMWSJMY6BU47A, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.858756961 (577h7m37.729840006s) [noise=true , address=145.79.227.13:4803 , piece_id=OS57GYKLQ563PPPGNEM47UPYEXUIO3GLTV2HH2PX4JJW6SJ33ARQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.866925414 (577h7m37.738008449s) [noise=true , address=145.79.227.9:4366 , piece_id=LD55E6A643OUSAP4GVRMUTUS4ZENGJQUX5ZSP5FK45AHADAPPOMA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.894357890 (577h7m37.765440925s) [noise=true , address=145.79.227.41:4085 , piece_id=GPCHKPBO7LZZHQ7EXRX5U4T6QWIND7YEAQAS6NQFIHTPEYNWAPNA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.115137568 (577h7m37.986220613s) [noise=true , address=45.91.116.13:4154 , piece_id=XX7NAO3AGJQZRAHBUBDJLJ7H3GD634FRKBNPWLTHQRTDI6OS3MNQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.246381235 (577h7m38.11746427s) [noise=true , address=45.91.116.6:4976 , piece_id=BP75OGDJNQ5QKWJXR5PAIFUF5SRS6RFNOX2AWETL72O2JLUQBM7A, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.427030097 (577h7m38.298113132s) [noise=true , address=45.91.116.4:4939 , piece_id=SQZSKGN6732LE3WMVHO6HVJ6VLK5VGUOGOBHIMVVHFJUW5FRNQNA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.810709200 (577h7m38.681792235s) [noise=true , address=77.90.31.16:4735 , piece_id=JQLZHLS2ATKE6ZF373M27CBACXR7SMXAHLGEMT7V4OHACAD5GY3Q, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.810862221 (577h7m38.681945266s) [seg_pos={0 0}, upload=505935]: Segment reached optimal threshold of 13 pieces. Dec 29 16:36:38.811163515 (577h7m38.68224658s) [noise=true , address=145.79.227.34:4905 , piece_id=IS52U3DJYFBPFVQTTGT5H6TR4NG5XKSJGW2DI3OBZJTLDZ5EL2GA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 16:36:38.811176659 (577h7m38.682259705s) [noise=true , address=145.79.227.34:4905 , piece_id=IS52U3DJYFBPFVQTTGT5H6TR4NG5XKSJGW2DI3OBZJTLDZ5EL2GA, seg_pos={0 0}, upload=505935]: Long tail cancellation due to optimal threshold Dec 29 16:36:38.811206516 (577h7m38.682289551s) [noise=true , address=145.79.227.6:4258 , piece_id=YRA6H2OGN6T67OZSAKY4UXPU3KSL3BPZRBRJL24GZVSAXUPKLDEA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:59834->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:59834->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59834->145.79.227.6:4258: use of closed network connection Dec 29 16:36:38.811239589 (577h7m38.682322625s) [noise=true , address=145.79.227.6:4258 , piece_id=YRA6H2OGN6T67OZSAKY4UXPU3KSL3BPZRBRJL24GZVSAXUPKLDEA, seg_pos={0 0}, upload=505935]: Long tail cancellation due to optimal threshold Dec 29 16:36:38.811510666 (577h7m38.682593711s) [noise=true , address=45.91.116.10:4084 , piece_id=XOMVJQE76AU3BNS5AVWIXUG6GHGAYVQT2VWAQ2N2MMI2PY6GISWQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 16:36:38.811520815 (577h7m38.68260386s) [noise=true , address=45.91.116.10:4084 , piece_id=XOMVJQE76AU3BNS5AVWIXUG6GHGAYVQT2VWAQ2N2MMI2PY6GISWQ, seg_pos={0 0}, upload=505935]: Long tail cancellation due to optimal threshold Dec 29 16:36:38.811545071 (577h7m38.682628116s) [seg_pos={0 0}, upload=505935]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:36:38.811713773 (577h7m38.682796858s) [upload=505935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:36:38.811721347 (577h7m38.682804412s) [upload=505935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:36:39.112152102 (577h7m38.983235137s) [upload=505935]: Done uploading segments... Dec 29 16:36:39.112171789 (577h7m38.983254834s) [upload=505935]: Upload finished. err: <nil> Duration: 5.968966186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X.X......X.....X......X...XX.XXX....X...XXXXX.X..XX...XX] duration: 30.004433456s trace: https://jaeger.dev.storj.io/trace/2b8bccfebf290739 FAIL (2025-12-29 16:37:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:36:35.608651446 (577h7m35.479734491s) [upload=505935]: Starting upload Dec 29 16:36:35.608732149 (577h7m35.479815184s) [upload=505935]: Done starting upload Dec 29 16:36:35.608844242 (577h7m35.479927277s) [upload=505935]: Uploading segments... Dec 29 16:36:35.608934424 (577h7m35.480017459s) [upload=505935]: Got next segment. Inline: false Dec 29 16:36:35.609020818 (577h7m35.480103853s) [upload=505935]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:36:35.609026500 (577h7m35.480109535s) [upload=505935]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:36:35.854751968 (577h7m35.725835003s) [seg_pos={0 0}, upload=505935]: Begin upload segment... Dec 29 16:36:35.854793076 (577h7m35.725876111s) [seg_pos={0 0}, upload=505935]: Done begin upload segment. Dec 29 16:36:35.854818184 (577h7m35.725901229s) [noise=true , address=145.79.227.13:4803 , piece_id=OS57GYKLQ563PPPGNEM47UPYEXUIO3GLTV2HH2PX4JJW6SJ33ARQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.854831449 (577h7m35.725914484s) [noise=true , address=45.91.116.13:4154 , piece_id=XX7NAO3AGJQZRAHBUBDJLJ7H3GD634FRKBNPWLTHQRTDI6OS3MNQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.855216492 (577h7m35.726299527s) [noise=true , address=145.79.227.51:4411 , piece_id=HNFX255R67I5MJ54O5PEHT4QFQ7KOIYURPT47EOFVA7SS2PCKQHA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.855502437 (577h7m35.726585472s) [noise=true , address=145.79.227.45:4231 , piece_id=T2UFIYKIHAMH36PH2QYTJNJ6H5VTZWHBJQRVLHIVTWSI4ZY4VHIA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.855778292 (577h7m35.726861327s) [noise=true , address=145.79.227.6:4258 , piece_id=YRA6H2OGN6T67OZSAKY4UXPU3KSL3BPZRBRJL24GZVSAXUPKLDEA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856054969 (577h7m35.727138004s) [noise=true , address=45.91.116.6:4976 , piece_id=BP75OGDJNQ5QKWJXR5PAIFUF5SRS6RFNOX2AWETL72O2JLUQBM7A, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856307390 (577h7m35.727390425s) [noise=true , address=145.79.227.26:4653 , piece_id=RFBCUXNRC4ME5255WBK24JS7POEWAYZ4GMIXGNREOIO2A5UYYBEQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856554701 (577h7m35.727637746s) [noise=true , address=145.79.227.42:4123 , piece_id=6554I3WSWJ6USI4XH5KTXVATOAL2DS623MZD5TZRMWSJMY6BU47A, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.856856014 (577h7m35.727939049s) [noise=true , address=77.90.31.16:4735 , piece_id=JQLZHLS2ATKE6ZF373M27CBACXR7SMXAHLGEMT7V4OHACAD5GY3Q, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.858488723 (577h7m35.729571768s) [noise=true , address=145.79.227.34:4905 , piece_id=IS52U3DJYFBPFVQTTGT5H6TR4NG5XKSJGW2DI3OBZJTLDZ5EL2GA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.858774286 (577h7m35.729857332s) [noise=true , address=145.79.227.9:4366 , piece_id=LD55E6A643OUSAP4GVRMUTUS4ZENGJQUX5ZSP5FK45AHADAPPOMA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.859827964 (577h7m35.730911009s) [noise=true , address=145.79.227.12:4765 , piece_id=2YVZUMYBBBHDNX3IUX4HHLRUPFDZJWMF5H44L3N3RR4YA4L3PZQA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.861024642 (577h7m35.732107777s) [noise=true , address=145.79.227.41:4085 , piece_id=GPCHKPBO7LZZHQ7EXRX5U4T6QWIND7YEAQAS6NQFIHTPEYNWAPNA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.862361888 (577h7m35.733444923s) [noise=true , address=45.91.116.4:4939 , piece_id=SQZSKGN6732LE3WMVHO6HVJ6VLK5VGUOGOBHIMVVHFJUW5FRNQNA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.863708903 (577h7m35.734791948s) [noise=true , address=145.79.227.50:4375 , piece_id=CMDNIPTQEFYQWBG3JLRXGML6FDTWCMU4GGSUVLDX7ZNQPZDSS7XA, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:35.865083391 (577h7m35.736166426s) [noise=true , address=45.91.116.10:4084 , piece_id=XOMVJQE76AU3BNS5AVWIXUG6GHGAYVQT2VWAQ2N2MMI2PY6GISWQ, seg_pos={0 0}, upload=505935]: Uploading piece... Dec 29 16:36:37.555310093 (577h7m37.426393138s) [upload=505935]: Next returned nil segment Dec 29 16:36:37.555330322 (577h7m37.426413357s) [upload=505935]: Waiting for error group managing segments... Dec 29 16:36:37.836205198 (577h7m37.707288233s) [noise=true , address=145.79.227.45:4231 , piece_id=T2UFIYKIHAMH36PH2QYTJNJ6H5VTZWHBJQRVLHIVTWSI4ZY4VHIA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.844111122 (577h7m37.715194157s) [noise=true , address=145.79.227.50:4375 , piece_id=CMDNIPTQEFYQWBG3JLRXGML6FDTWCMU4GGSUVLDX7ZNQPZDSS7XA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.844526874 (577h7m37.715609919s) [noise=true , address=145.79.227.26:4653 , piece_id=RFBCUXNRC4ME5255WBK24JS7POEWAYZ4GMIXGNREOIO2A5UYYBEQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.849170607 (577h7m37.720253652s) [noise=true , address=145.79.227.51:4411 , piece_id=HNFX255R67I5MJ54O5PEHT4QFQ7KOIYURPT47EOFVA7SS2PCKQHA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.850101289 (577h7m37.721184324s) [noise=true , address=145.79.227.12:4765 , piece_id=2YVZUMYBBBHDNX3IUX4HHLRUPFDZJWMF5H44L3N3RR4YA4L3PZQA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.858234125 (577h7m37.72931716s) [noise=true , address=145.79.227.42:4123 , piece_id=6554I3WSWJ6USI4XH5KTXVATOAL2DS623MZD5TZRMWSJMY6BU47A, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.858756961 (577h7m37.729840006s) [noise=true , address=145.79.227.13:4803 , piece_id=OS57GYKLQ563PPPGNEM47UPYEXUIO3GLTV2HH2PX4JJW6SJ33ARQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.866925414 (577h7m37.738008449s) [noise=true , address=145.79.227.9:4366 , piece_id=LD55E6A643OUSAP4GVRMUTUS4ZENGJQUX5ZSP5FK45AHADAPPOMA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:37.894357890 (577h7m37.765440925s) [noise=true , address=145.79.227.41:4085 , piece_id=GPCHKPBO7LZZHQ7EXRX5U4T6QWIND7YEAQAS6NQFIHTPEYNWAPNA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.115137568 (577h7m37.986220613s) [noise=true , address=45.91.116.13:4154 , piece_id=XX7NAO3AGJQZRAHBUBDJLJ7H3GD634FRKBNPWLTHQRTDI6OS3MNQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.246381235 (577h7m38.11746427s) [noise=true , address=45.91.116.6:4976 , piece_id=BP75OGDJNQ5QKWJXR5PAIFUF5SRS6RFNOX2AWETL72O2JLUQBM7A, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.427030097 (577h7m38.298113132s) [noise=true , address=45.91.116.4:4939 , piece_id=SQZSKGN6732LE3WMVHO6HVJ6VLK5VGUOGOBHIMVVHFJUW5FRNQNA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.810709200 (577h7m38.681792235s) [noise=true , address=77.90.31.16:4735 , piece_id=JQLZHLS2ATKE6ZF373M27CBACXR7SMXAHLGEMT7V4OHACAD5GY3Q, seg_pos={0 0}, upload=505935]: Done uploading piece. err: <nil> Dec 29 16:36:38.810862221 (577h7m38.681945266s) [seg_pos={0 0}, upload=505935]: Segment reached optimal threshold of 13 pieces. Dec 29 16:36:38.811163515 (577h7m38.68224658s) [noise=true , address=145.79.227.34:4905 , piece_id=IS52U3DJYFBPFVQTTGT5H6TR4NG5XKSJGW2DI3OBZJTLDZ5EL2GA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 16:36:38.811176659 (577h7m38.682259705s) [noise=true , address=145.79.227.34:4905 , piece_id=IS52U3DJYFBPFVQTTGT5H6TR4NG5XKSJGW2DI3OBZJTLDZ5EL2GA, seg_pos={0 0}, upload=505935]: Long tail cancellation due to optimal threshold Dec 29 16:36:38.811206516 (577h7m38.682289551s) [noise=true , address=145.79.227.6:4258 , piece_id=YRA6H2OGN6T67OZSAKY4UXPU3KSL3BPZRBRJL24GZVSAXUPKLDEA, seg_pos={0 0}, upload=505935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:59834->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:59834->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59834->145.79.227.6:4258: use of closed network connection Dec 29 16:36:38.811239589 (577h7m38.682322625s) [noise=true , address=145.79.227.6:4258 , piece_id=YRA6H2OGN6T67OZSAKY4UXPU3KSL3BPZRBRJL24GZVSAXUPKLDEA, seg_pos={0 0}, upload=505935]: Long tail cancellation due to optimal threshold Dec 29 16:36:38.811510666 (577h7m38.682593711s) [noise=true , address=45.91.116.10:4084 , piece_id=XOMVJQE76AU3BNS5AVWIXUG6GHGAYVQT2VWAQ2N2MMI2PY6GISWQ, seg_pos={0 0}, upload=505935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 16:36:38.811520815 (577h7m38.68260386s) [noise=true , address=45.91.116.10:4084 , piece_id=XOMVJQE76AU3BNS5AVWIXUG6GHGAYVQT2VWAQ2N2MMI2PY6GISWQ, seg_pos={0 0}, upload=505935]: Long tail cancellation due to optimal threshold Dec 29 16:36:38.811545071 (577h7m38.682628116s) [seg_pos={0 0}, upload=505935]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:36:38.811713773 (577h7m38.682796858s) [upload=505935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:36:38.811721347 (577h7m38.682804412s) [upload=505935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:36:39.112152102 (577h7m38.983235137s) [upload=505935]: Done uploading segments... Dec 29 16:36:39.112171789 (577h7m38.983254834s) [upload=505935]: Upload finished. err: <nil> Duration: 30.004457001s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XXX] duration: 5.000959984s trace: https://jaeger.dev.storj.io/trace/56715990b227e1e6 FAIL (2025-12-29 16:36:34 UTC) orbiter warning: context deadline exceeded Duration: 5.000982356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X....X.X......X.....X......X...XX.XXX....X...XXXXX.X..XX...X] duration: 30.003454462s trace: https://jaeger.dev.storj.io/trace/6a176f25e569bccc FAIL (2025-12-29 16:36:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:35:45.241414840 (577h6m45.112497885s) [upload=505923]: Starting upload Dec 29 16:35:45.241492417 (577h6m45.112575452s) [upload=505923]: Done starting upload Dec 29 16:35:45.241646180 (577h6m45.112729215s) [upload=505923]: Uploading segments... Dec 29 16:35:45.241752944 (577h6m45.112835979s) [upload=505923]: Got next segment. Inline: false Dec 29 16:35:45.241829459 (577h6m45.112912504s) [upload=505923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:35:45.241836452 (577h6m45.112919507s) [upload=505923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:35:45.486225184 (577h6m45.357308229s) [seg_pos={0 0}, upload=505923]: Begin upload segment... Dec 29 16:35:45.486338369 (577h6m45.357421404s) [seg_pos={0 0}, upload=505923]: Done begin upload segment. Dec 29 16:35:45.486395589 (577h6m45.357478634s) [noise=true , address=145.79.227.36:4518 , piece_id=RAV2PCMNAF2EPQAGOES3G2EURCSZ5FUIGQOVAJFRFSK7K4IHI4YQ, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.486421026 (577h6m45.357504061s) [noise=true , address=77.90.31.18:4808 , piece_id=XD2PVODX7RNIERHHTCULSFOJZANS7JA7JQFU4GWUBTZ4NX7CRLOQ, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.486959212 (577h6m45.358042247s) [noise=true , address=145.79.227.22:4510 , piece_id=6FMPAHUGKDTYYTVXVIDS3EI5R575E33Y2Q3YJUJ2W3ON6OW6EK6Q, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.487293708 (577h6m45.358376743s) [noise=true , address=145.79.227.52:4445 , piece_id=3K7SYX5BWE6XRGSV72XN2JHFHXXTGDDRKER5VMCP6JI6GRXMD4DQ, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.487946561 (577h6m45.359029597s) [noise=true , address=77.90.31.19:4842 , piece_id=MU4YXMJVPOEH5RBYD2L7OMQG6XYMVLVIM7DS3VBHFEASOUSYLARQ, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.488511667 (577h6m45.359594712s) [noise=true , address=145.79.227.34:4905 , piece_id=DQ4WAV7FPRWZXBWDORL6ILC7DRRPKTTQB2KUD43ZFN3YYKPLOMAA, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.488956955 (577h6m45.36004s) [noise=true , address=77.90.31.14:4664 , piece_id=VUN5OWKFJOBEBKSZAAFQX4F3HE55BQY5WASSVAVKU3CAUB2WQQQA, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.490905917 (577h6m45.361988952s) [noise=true , address=145.79.227.37:4566 , piece_id=BXNYKSVQVRGTBA6FIHUZMB6I2BJNHWTMQMFBDCG3GWXQR5PSNP5Q, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.505227658 (577h6m45.376310693s) [noise=true , address=45.91.116.13:4155 , piece_id=SIOTA7BDRPX7HEEKF7CRYIZLEFA7ELFD23AQSWUW6CFP7OOINIQA, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.506505210 (577h6m45.377588255s) [noise=true , address=145.79.227.32:4836 , piece_id=5NPDJQONHF2WVSJSQGIG2TQBTIUJNDSWHLIU7QSP2EWBC2X3MEVQ, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.506697757 (577h6m45.377780792s) [noise=true , address=45.91.116.10:4083 , piece_id=VHQRVZPV2OYFCRXNGXIC5QHHI5TAVGSAWGCG4HZLD6PV4B2XHUGQ, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.508832612 (577h6m45.379915647s) [noise=true , address=145.79.227.46:4603 , piece_id=ZUVTJCHQHARHTANGBCVM5KWUKKND5SZ4RLZEYIJA33VGYUEM2HDA, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.510428641 (577h6m45.381511676s) [noise=true , address=45.91.116.12:4227 , piece_id=2QKXXB2V5Y2UTAKR22UGLCLCZYIFCAYR7C6BBEAEP6XEMUPZCCHA, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.512908533 (577h6m45.383991578s) [noise=true , address=45.91.116.8:4047 , piece_id=ENHN7YLQ6TIWR3BQ2RCHRH3U3KH7HKJHOWNH5QJAT5JU757OQISQ, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.515020865 (577h6m45.38610391s) [noise=true , address=77.90.31.17:4770 , piece_id=FJDVYSXOMWG6RPXAVX7UYYIE6CXWUYWELCVNOZZHUVLCUXGLKUQQ, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:45.516348514 (577h6m45.387431549s) [noise=true , address=145.79.227.15:4876 , piece_id=O2XJ6TEWP4SUUSLJME2US7EPEK4MGLE474BWNKDQJGM3CQFJR6JQ, seg_pos={0 0}, upload=505923]: Uploading piece... Dec 29 16:35:47.129562663 (577h6m47.000645698s) [upload=505923]: Next returned nil segment Dec 29 16:35:47.129596768 (577h6m47.000679803s) [upload=505923]: Waiting for error group managing segments... Dec 29 16:35:47.467460932 (577h6m47.338543967s) [noise=true , address=145.79.227.36:4518 , piece_id=RAV2PCMNAF2EPQAGOES3G2EURCSZ5FUIGQOVAJFRFSK7K4IHI4YQ, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:47.476792641 (577h6m47.347875676s) [noise=true , address=145.79.227.52:4445 , piece_id=3K7SYX5BWE6XRGSV72XN2JHFHXXTGDDRKER5VMCP6JI6GRXMD4DQ, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:47.488041910 (577h6m47.359124945s) [noise=true , address=145.79.227.22:4510 , piece_id=6FMPAHUGKDTYYTVXVIDS3EI5R575E33Y2Q3YJUJ2W3ON6OW6EK6Q, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:47.497961970 (577h6m47.369045005s) [noise=true , address=145.79.227.37:4566 , piece_id=BXNYKSVQVRGTBA6FIHUZMB6I2BJNHWTMQMFBDCG3GWXQR5PSNP5Q, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:47.504037398 (577h6m47.375120433s) [noise=true , address=145.79.227.15:4876 , piece_id=O2XJ6TEWP4SUUSLJME2US7EPEK4MGLE474BWNKDQJGM3CQFJR6JQ, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:47.511797975 (577h6m47.38288102s) [noise=true , address=145.79.227.32:4836 , piece_id=5NPDJQONHF2WVSJSQGIG2TQBTIUJNDSWHLIU7QSP2EWBC2X3MEVQ, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:47.517296656 (577h6m47.388379691s) [noise=true , address=145.79.227.46:4603 , piece_id=ZUVTJCHQHARHTANGBCVM5KWUKKND5SZ4RLZEYIJA33VGYUEM2HDA, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:47.645921627 (577h6m47.517004662s) [noise=true , address=77.90.31.19:4842 , piece_id=MU4YXMJVPOEH5RBYD2L7OMQG6XYMVLVIM7DS3VBHFEASOUSYLARQ, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:47.771307791 (577h6m47.642390836s) [noise=true , address=77.90.31.14:4664 , piece_id=VUN5OWKFJOBEBKSZAAFQX4F3HE55BQY5WASSVAVKU3CAUB2WQQQA, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:47.807085627 (577h6m47.678168662s) [noise=true , address=45.91.116.13:4155 , piece_id=SIOTA7BDRPX7HEEKF7CRYIZLEFA7ELFD23AQSWUW6CFP7OOINIQA, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:47.875811803 (577h6m47.746894838s) [noise=true , address=45.91.116.8:4047 , piece_id=ENHN7YLQ6TIWR3BQ2RCHRH3U3KH7HKJHOWNH5QJAT5JU757OQISQ, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:47.953510352 (577h6m47.824593397s) [noise=true , address=45.91.116.12:4227 , piece_id=2QKXXB2V5Y2UTAKR22UGLCLCZYIFCAYR7C6BBEAEP6XEMUPZCCHA, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:48.070593245 (577h6m47.94167628s) [noise=true , address=77.90.31.18:4808 , piece_id=XD2PVODX7RNIERHHTCULSFOJZANS7JA7JQFU4GWUBTZ4NX7CRLOQ, seg_pos={0 0}, upload=505923]: Done uploading piece. err: <nil> Dec 29 16:35:48.070633632 (577h6m47.941716667s) [seg_pos={0 0}, upload=505923]: Segment reached optimal threshold of 13 pieces. Dec 29 16:35:48.070821089 (577h6m47.941904124s) [noise=true , address=45.91.116.10:4083 , piece_id=VHQRVZPV2OYFCRXNGXIC5QHHI5TAVGSAWGCG4HZLD6PV4B2XHUGQ, seg_pos={0 0}, upload=505923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 16:35:48.070827661 (577h6m47.941910696s) [noise=true , address=45.91.116.10:4083 , piece_id=VHQRVZPV2OYFCRXNGXIC5QHHI5TAVGSAWGCG4HZLD6PV4B2XHUGQ, seg_pos={0 0}, upload=505923]: Long tail cancellation due to optimal threshold Dec 29 16:35:48.071054694 (577h6m47.942137739s) [noise=true , address=145.79.227.34:4905 , piece_id=DQ4WAV7FPRWZXBWDORL6ILC7DRRPKTTQB2KUD43ZFN3YYKPLOMAA, seg_pos={0 0}, upload=505923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 16:35:48.071094520 (577h6m47.942177555s) [noise=true , address=145.79.227.34:4905 , piece_id=DQ4WAV7FPRWZXBWDORL6ILC7DRRPKTTQB2KUD43ZFN3YYKPLOMAA, seg_pos={0 0}, upload=505923]: Long tail cancellation due to optimal threshold Dec 29 16:35:48.071185182 (577h6m47.942268227s) [noise=true , address=77.90.31.17:4770 , piece_id=FJDVYSXOMWG6RPXAVX7UYYIE6CXWUYWELCVNOZZHUVLCUXGLKUQQ, seg_pos={0 0}, upload=505923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 16:35:48.071200311 (577h6m47.942283346s) [noise=true , address=77.90.31.17:4770 , piece_id=FJDVYSXOMWG6RPXAVX7UYYIE6CXWUYWELCVNOZZHUVLCUXGLKUQQ, seg_pos={0 0}, upload=505923]: Long tail cancellation due to optimal threshold Dec 29 16:35:48.071227804 (577h6m47.942310859s) [seg_pos={0 0}, upload=505923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:35:48.071358382 (577h6m47.942441427s) [upload=505923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:35:48.071366648 (577h6m47.942449683s) [upload=505923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:35:48.366412769 (577h6m48.237495804s) [upload=505923]: Done uploading segments... Dec 29 16:35:48.366429411 (577h6m48.237512446s) [upload=505923]: Upload finished. err: <nil> Duration: 30.003478828s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 15.409308255s trace: https://jaeger.dev.storj.io/trace/25eb57d21bedad4d FAIL (2025-12-29 16:35:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, Piece ID: KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA) context deadline exceeded; piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ) context deadline exceeded; piecestore: (Node ID: 12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, Piece ID: MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA) context deadline exceeded ======LOGS====== 530948251s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660301397 (577h2m12.531384432s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660322538 (577h2m12.531405583s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660796670 (577h2m12.531879705s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661151005 (577h2m12.53223404s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661355344 (577h2m12.532438389s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661875545 (577h2m12.53295859s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662039136 (577h2m12.533122202s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662350129 (577h2m12.533433174s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662716055 (577h2m12.5337991s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662805526 (577h2m12.533888571s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.663913205 (577h2m12.53499624s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.664659345 (577h2m12.535742411s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.666957802 (577h2m12.538040847s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.668348230 (577h2m12.539431305s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:14.044161823 (577h2m13.915244868s) [upload=505855]: Next returned nil segment Dec 29 16:31:14.044175078 (577h2m13.915258113s) [upload=505855]: Waiting for error group managing segments... Dec 29 16:31:14.378564616 (577h2m14.249647661s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.513685195 (577h2m14.384768231s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.765759753 (577h2m14.636842788s) [noise=true , address=45.91.116.5:4316 , piece_id=BVVWSLENSRBMD5OBFJHPOBKDYHE55LLKPQ3MNLD4DCBPAKPUTGTA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.796869410 (577h2m14.667952445s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.805110142 (577h2m14.676193187s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.809656709 (577h2m14.680739744s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.828088656 (577h2m14.699171701s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.916522628 (577h2m14.787605663s) [noise=true , address=77.90.31.17:4795 , piece_id=QXXJ6VOBDRVCVRRMLSBQO7MB2DS5PUMDE7KPGOIJ77AJOJPSU7XQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.387028071 (577h2m15.258111106s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.424600454 (577h2m15.295683489s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.431096903 (577h2m15.302179939s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.454782875 (577h2m15.32586591s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.845291312 (577h2m15.716374357s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302138686 (577h2m16.173221731s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302163113 (577h2m16.173246148s) [seg_pos={0 0}, upload=505855]: Segment reached optimal threshold of 14 pieces. Dec 29 16:31:16.302359598 (577h2m16.173442663s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XmqDfLfXDjJ6Bo9PAYiVgshEBPnfQ1D9yC6CcvXpPRCWq4Wrs): protocol: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection Dec 29 16:31:16.302370097 (577h2m16.173453152s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302392490 (577h2m16.173475535s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection Dec 29 16:31:16.302401277 (577h2m16.173484322s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302574897 (577h2m16.173657952s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302582952 (577h2m16.173666018s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302723921 (577h2m16.173806956s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302731816 (577h2m16.173814851s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302750131 (577h2m16.173833176s) [seg_pos={0 0}, upload=505855]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:31:16.302842045 (577h2m16.17392509s) [upload=505855]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:31:16.302848428 (577h2m16.173931473s) [upload=505855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:31:16.589365678 (577h2m16.460448733s) [upload=505855]: Done uploading segments... Dec 29 16:31:16.589388892 (577h2m16.460471938s) [upload=505855]: Upload finished. err: <nil> Duration: 15.409338172s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..XX] duration: 5.001042485s trace: https://jaeger.dev.storj.io/trace/55f822dc15b1e74 FAIL (2025-12-29 16:35:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001065781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X....X.X......X.....X......X...XX.XXX....X...XXXXX.X..XX...] duration: 15.802506905s trace: https://jaeger.dev.storj.io/trace/24520496a51f996 FAIL (2025-12-29 16:35:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:33:05.637966222 (577h4m5.509049257s) [upload=505882]: Starting upload Dec 29 16:33:05.638021567 (577h4m5.509104602s) [upload=505882]: Done starting upload Dec 29 16:33:05.638067174 (577h4m5.50915021s) [upload=505882]: Uploading segments... Dec 29 16:33:05.638221718 (577h4m5.509304763s) [upload=505882]: Got next segment. Inline: false Dec 29 16:33:05.638253910 (577h4m5.509336945s) [upload=505882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:33:05.638258649 (577h4m5.509341684s) [upload=505882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:33:05.893573608 (577h4m5.764656643s) [seg_pos={0 0}, upload=505882]: Begin upload segment... Dec 29 16:33:05.893622982 (577h4m5.764706027s) [seg_pos={0 0}, upload=505882]: Done begin upload segment. Dec 29 16:33:05.893656255 (577h4m5.76473929s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.893669259 (577h4m5.764752294s) [noise=true , address=77.90.31.10:4519 , piece_id=M7CIKY6NNGADTISYWYNPP2YVJGGNOAXREIIGO4RHR54DZUW6YLUA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.894660668 (577h4m5.765743713s) [noise=true , address=145.79.227.5:4221 , piece_id=2SZWF3GDUPSQ5C6KXXCZYHNWCCMBY5UEASL4JNEW7LWHPDNPLGNQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.894728627 (577h4m5.765811672s) [noise=true , address=145.79.227.9:4366 , piece_id=ZI234HLBZR6H5XZYOB4ZX2GGPCE2Q5AQJXEQVIDIXO3BZ3RKZ6WA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.895460391 (577h4m5.766543436s) [noise=true , address=145.79.227.21:4475 , piece_id=GKUGCRAMQNBWNKD5TEDT7MXCW76Y5X6X264Z4WPTP7SCZGZPHAQA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.895854220 (577h4m5.766937255s) [noise=true , address=45.91.116.14:4191 , piece_id=C46MUQ5WXGG4F3WYNBQH3YBU4DXXXST2FHIKKPJ3SLFIK72UFIDA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.897017725 (577h4m5.76810077s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.897077018 (577h4m5.768160053s) [noise=true , address=77.90.31.19:4841 , piece_id=U2GDKURYAD6DBCDECSOXDKWKXKSN4LQ6MPHO3RO2H6KZAUJSCU5Q, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898015766 (577h4m5.769098801s) [noise=true , address=45.91.116.8:4047 , piece_id=MG4FDE5VWDEUIPNM4RZIXF3MVIJSESA7FUL42OMZTULOBV7VZB5A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898475862 (577h4m5.769558907s) [noise=true , address=145.79.227.6:4257 , piece_id=ZZJO76O6VOS7AYUSUJC3M422ZLNMEWQQ5FNRASAKCIH7QR6FKPBQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898971546 (577h4m5.770054581s) [noise=true , address=145.79.227.4:4186 , piece_id=O3XGN5IFZUWRAN6I4BEKOVPSJQJIETNNTNWR3FII3JLW46SIUCZQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899017404 (577h4m5.770100439s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYS6BRKSFUAY4PPEMBS3YKIH4RDTRXZ24QY5FNFJCUXVVCDA6HRQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899659075 (577h4m5.77074211s) [noise=true , address=145.79.227.7:4294 , piece_id=A4ACPZJB6KYLQQET6FYQPLX4SFYWNTOVC7PEWNDYRPIPYKRFAY5A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899723157 (577h4m5.770806202s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.900184796 (577h4m5.771267831s) [noise=true , address=77.90.31.16:4736 , piece_id=7VY4YJ3GBZVSJEGKI5IZOWOGUJOR33EOLF6AUF67VY5GCVG7BFLA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.900859971 (577h4m5.771943026s) [noise=true , address=145.79.227.11:4731 , piece_id=AD5NLKAMUYAD7CEUKLBZGPJWWRAE7PGR5QMFZ3HF3TXJSILZ6JHA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:07.586370754 (577h4m7.45745379s) [upload=505882]: Next returned nil segment Dec 29 16:33:07.586382908 (577h4m7.457465943s) [upload=505882]: Waiting for error group managing segments... Dec 29 16:33:07.871908942 (577h4m7.742991977s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYS6BRKSFUAY4PPEMBS3YKIH4RDTRXZ24QY5FNFJCUXVVCDA6HRQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.882984622 (577h4m7.754067657s) [noise=true , address=145.79.227.6:4257 , piece_id=ZZJO76O6VOS7AYUSUJC3M422ZLNMEWQQ5FNRASAKCIH7QR6FKPBQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.885295252 (577h4m7.756378297s) [noise=true , address=145.79.227.5:4221 , piece_id=2SZWF3GDUPSQ5C6KXXCZYHNWCCMBY5UEASL4JNEW7LWHPDNPLGNQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.902485234 (577h4m7.773568279s) [noise=true , address=145.79.227.11:4731 , piece_id=AD5NLKAMUYAD7CEUKLBZGPJWWRAE7PGR5QMFZ3HF3TXJSILZ6JHA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.904503718 (577h4m7.775586763s) [noise=true , address=145.79.227.4:4186 , piece_id=O3XGN5IFZUWRAN6I4BEKOVPSJQJIETNNTNWR3FII3JLW46SIUCZQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.906260594 (577h4m7.777343629s) [noise=true , address=145.79.227.9:4366 , piece_id=ZI234HLBZR6H5XZYOB4ZX2GGPCE2Q5AQJXEQVIDIXO3BZ3RKZ6WA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.909545378 (577h4m7.780628413s) [noise=true , address=145.79.227.21:4475 , piece_id=GKUGCRAMQNBWNKD5TEDT7MXCW76Y5X6X264Z4WPTP7SCZGZPHAQA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.987049296 (577h4m7.858132331s) [noise=true , address=145.79.227.7:4294 , piece_id=A4ACPZJB6KYLQQET6FYQPLX4SFYWNTOVC7PEWNDYRPIPYKRFAY5A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.080346628 (577h4m7.951429664s) [noise=true , address=77.90.31.19:4841 , piece_id=U2GDKURYAD6DBCDECSOXDKWKXKSN4LQ6MPHO3RO2H6KZAUJSCU5Q, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.108083083 (577h4m7.979166118s) [noise=true , address=45.91.116.8:4047 , piece_id=MG4FDE5VWDEUIPNM4RZIXF3MVIJSESA7FUL42OMZTULOBV7VZB5A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.143935421 (577h4m8.015018456s) [noise=true , address=45.91.116.14:4191 , piece_id=C46MUQ5WXGG4F3WYNBQH3YBU4DXXXST2FHIKKPJ3SLFIK72UFIDA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.873452359 (577h4m8.744535394s) [noise=true , address=77.90.31.16:4736 , piece_id=7VY4YJ3GBZVSJEGKI5IZOWOGUJOR33EOLF6AUF67VY5GCVG7BFLA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:09.018634499 (577h4m8.889717534s) [noise=true , address=77.90.31.10:4519 , piece_id=M7CIKY6NNGADTISYWYNPP2YVJGGNOAXREIIGO4RHR54DZUW6YLUA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:09.018657852 (577h4m8.889740898s) [seg_pos={0 0}, upload=505882]: Segment reached optimal threshold of 13 pieces. Dec 29 16:33:09.018945279 (577h4m8.890028325s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.018970448 (577h4m8.890053493s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019101878 (577h4m8.890184923s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.019107689 (577h4m8.890190724s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019140251 (577h4m8.890223286s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.019171601 (577h4m8.890254646s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019194204 (577h4m8.890277249s) [seg_pos={0 0}, upload=505882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:33:09.019393022 (577h4m8.890476057s) [upload=505882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:33:09.019402090 (577h4m8.890485125s) [upload=505882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:33:09.318770071 (577h4m9.189853106s) [upload=505882]: Done uploading segments... Dec 29 16:33:09.318794417 (577h4m9.189877452s) [upload=505882]: Upload finished. err: <nil> Duration: 15.802535499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 48.30515726s trace: https://jaeger.dev.storj.io/trace/25eb57d21bedad4d FAIL (2025-12-29 16:35:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, Piece ID: KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA) context deadline exceeded; piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ) context deadline exceeded; piecestore: (Node ID: 12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, Piece ID: MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA) context deadline exceeded ======LOGS====== 530948251s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660301397 (577h2m12.531384432s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660322538 (577h2m12.531405583s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660796670 (577h2m12.531879705s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661151005 (577h2m12.53223404s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661355344 (577h2m12.532438389s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661875545 (577h2m12.53295859s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662039136 (577h2m12.533122202s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662350129 (577h2m12.533433174s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662716055 (577h2m12.5337991s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662805526 (577h2m12.533888571s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.663913205 (577h2m12.53499624s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.664659345 (577h2m12.535742411s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.666957802 (577h2m12.538040847s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.668348230 (577h2m12.539431305s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:14.044161823 (577h2m13.915244868s) [upload=505855]: Next returned nil segment Dec 29 16:31:14.044175078 (577h2m13.915258113s) [upload=505855]: Waiting for error group managing segments... Dec 29 16:31:14.378564616 (577h2m14.249647661s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.513685195 (577h2m14.384768231s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.765759753 (577h2m14.636842788s) [noise=true , address=45.91.116.5:4316 , piece_id=BVVWSLENSRBMD5OBFJHPOBKDYHE55LLKPQ3MNLD4DCBPAKPUTGTA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.796869410 (577h2m14.667952445s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.805110142 (577h2m14.676193187s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.809656709 (577h2m14.680739744s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.828088656 (577h2m14.699171701s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.916522628 (577h2m14.787605663s) [noise=true , address=77.90.31.17:4795 , piece_id=QXXJ6VOBDRVCVRRMLSBQO7MB2DS5PUMDE7KPGOIJ77AJOJPSU7XQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.387028071 (577h2m15.258111106s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.424600454 (577h2m15.295683489s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.431096903 (577h2m15.302179939s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.454782875 (577h2m15.32586591s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.845291312 (577h2m15.716374357s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302138686 (577h2m16.173221731s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302163113 (577h2m16.173246148s) [seg_pos={0 0}, upload=505855]: Segment reached optimal threshold of 14 pieces. Dec 29 16:31:16.302359598 (577h2m16.173442663s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XmqDfLfXDjJ6Bo9PAYiVgshEBPnfQ1D9yC6CcvXpPRCWq4Wrs): protocol: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection Dec 29 16:31:16.302370097 (577h2m16.173453152s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302392490 (577h2m16.173475535s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection Dec 29 16:31:16.302401277 (577h2m16.173484322s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302574897 (577h2m16.173657952s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302582952 (577h2m16.173666018s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302723921 (577h2m16.173806956s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302731816 (577h2m16.173814851s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302750131 (577h2m16.173833176s) [seg_pos={0 0}, upload=505855]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:31:16.302842045 (577h2m16.17392509s) [upload=505855]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:31:16.302848428 (577h2m16.173931473s) [upload=505855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:31:16.589365678 (577h2m16.460448733s) [upload=505855]: Done uploading segments... Dec 29 16:31:16.589388892 (577h2m16.460471938s) [upload=505855]: Upload finished. err: <nil> Duration: 48.305186225s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X....X.X......X.....X......X...XX.XXX....X...XXXXX.X..XX..] duration: 28.818151099s trace: https://jaeger.dev.storj.io/trace/24520496a51f996 FAIL (2025-12-29 16:35:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:33:05.637966222 (577h4m5.509049257s) [upload=505882]: Starting upload Dec 29 16:33:05.638021567 (577h4m5.509104602s) [upload=505882]: Done starting upload Dec 29 16:33:05.638067174 (577h4m5.50915021s) [upload=505882]: Uploading segments... Dec 29 16:33:05.638221718 (577h4m5.509304763s) [upload=505882]: Got next segment. Inline: false Dec 29 16:33:05.638253910 (577h4m5.509336945s) [upload=505882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:33:05.638258649 (577h4m5.509341684s) [upload=505882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:33:05.893573608 (577h4m5.764656643s) [seg_pos={0 0}, upload=505882]: Begin upload segment... Dec 29 16:33:05.893622982 (577h4m5.764706027s) [seg_pos={0 0}, upload=505882]: Done begin upload segment. Dec 29 16:33:05.893656255 (577h4m5.76473929s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.893669259 (577h4m5.764752294s) [noise=true , address=77.90.31.10:4519 , piece_id=M7CIKY6NNGADTISYWYNPP2YVJGGNOAXREIIGO4RHR54DZUW6YLUA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.894660668 (577h4m5.765743713s) [noise=true , address=145.79.227.5:4221 , piece_id=2SZWF3GDUPSQ5C6KXXCZYHNWCCMBY5UEASL4JNEW7LWHPDNPLGNQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.894728627 (577h4m5.765811672s) [noise=true , address=145.79.227.9:4366 , piece_id=ZI234HLBZR6H5XZYOB4ZX2GGPCE2Q5AQJXEQVIDIXO3BZ3RKZ6WA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.895460391 (577h4m5.766543436s) [noise=true , address=145.79.227.21:4475 , piece_id=GKUGCRAMQNBWNKD5TEDT7MXCW76Y5X6X264Z4WPTP7SCZGZPHAQA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.895854220 (577h4m5.766937255s) [noise=true , address=45.91.116.14:4191 , piece_id=C46MUQ5WXGG4F3WYNBQH3YBU4DXXXST2FHIKKPJ3SLFIK72UFIDA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.897017725 (577h4m5.76810077s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.897077018 (577h4m5.768160053s) [noise=true , address=77.90.31.19:4841 , piece_id=U2GDKURYAD6DBCDECSOXDKWKXKSN4LQ6MPHO3RO2H6KZAUJSCU5Q, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898015766 (577h4m5.769098801s) [noise=true , address=45.91.116.8:4047 , piece_id=MG4FDE5VWDEUIPNM4RZIXF3MVIJSESA7FUL42OMZTULOBV7VZB5A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898475862 (577h4m5.769558907s) [noise=true , address=145.79.227.6:4257 , piece_id=ZZJO76O6VOS7AYUSUJC3M422ZLNMEWQQ5FNRASAKCIH7QR6FKPBQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898971546 (577h4m5.770054581s) [noise=true , address=145.79.227.4:4186 , piece_id=O3XGN5IFZUWRAN6I4BEKOVPSJQJIETNNTNWR3FII3JLW46SIUCZQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899017404 (577h4m5.770100439s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYS6BRKSFUAY4PPEMBS3YKIH4RDTRXZ24QY5FNFJCUXVVCDA6HRQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899659075 (577h4m5.77074211s) [noise=true , address=145.79.227.7:4294 , piece_id=A4ACPZJB6KYLQQET6FYQPLX4SFYWNTOVC7PEWNDYRPIPYKRFAY5A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899723157 (577h4m5.770806202s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.900184796 (577h4m5.771267831s) [noise=true , address=77.90.31.16:4736 , piece_id=7VY4YJ3GBZVSJEGKI5IZOWOGUJOR33EOLF6AUF67VY5GCVG7BFLA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.900859971 (577h4m5.771943026s) [noise=true , address=145.79.227.11:4731 , piece_id=AD5NLKAMUYAD7CEUKLBZGPJWWRAE7PGR5QMFZ3HF3TXJSILZ6JHA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:07.586370754 (577h4m7.45745379s) [upload=505882]: Next returned nil segment Dec 29 16:33:07.586382908 (577h4m7.457465943s) [upload=505882]: Waiting for error group managing segments... Dec 29 16:33:07.871908942 (577h4m7.742991977s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYS6BRKSFUAY4PPEMBS3YKIH4RDTRXZ24QY5FNFJCUXVVCDA6HRQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.882984622 (577h4m7.754067657s) [noise=true , address=145.79.227.6:4257 , piece_id=ZZJO76O6VOS7AYUSUJC3M422ZLNMEWQQ5FNRASAKCIH7QR6FKPBQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.885295252 (577h4m7.756378297s) [noise=true , address=145.79.227.5:4221 , piece_id=2SZWF3GDUPSQ5C6KXXCZYHNWCCMBY5UEASL4JNEW7LWHPDNPLGNQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.902485234 (577h4m7.773568279s) [noise=true , address=145.79.227.11:4731 , piece_id=AD5NLKAMUYAD7CEUKLBZGPJWWRAE7PGR5QMFZ3HF3TXJSILZ6JHA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.904503718 (577h4m7.775586763s) [noise=true , address=145.79.227.4:4186 , piece_id=O3XGN5IFZUWRAN6I4BEKOVPSJQJIETNNTNWR3FII3JLW46SIUCZQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.906260594 (577h4m7.777343629s) [noise=true , address=145.79.227.9:4366 , piece_id=ZI234HLBZR6H5XZYOB4ZX2GGPCE2Q5AQJXEQVIDIXO3BZ3RKZ6WA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.909545378 (577h4m7.780628413s) [noise=true , address=145.79.227.21:4475 , piece_id=GKUGCRAMQNBWNKD5TEDT7MXCW76Y5X6X264Z4WPTP7SCZGZPHAQA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.987049296 (577h4m7.858132331s) [noise=true , address=145.79.227.7:4294 , piece_id=A4ACPZJB6KYLQQET6FYQPLX4SFYWNTOVC7PEWNDYRPIPYKRFAY5A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.080346628 (577h4m7.951429664s) [noise=true , address=77.90.31.19:4841 , piece_id=U2GDKURYAD6DBCDECSOXDKWKXKSN4LQ6MPHO3RO2H6KZAUJSCU5Q, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.108083083 (577h4m7.979166118s) [noise=true , address=45.91.116.8:4047 , piece_id=MG4FDE5VWDEUIPNM4RZIXF3MVIJSESA7FUL42OMZTULOBV7VZB5A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.143935421 (577h4m8.015018456s) [noise=true , address=45.91.116.14:4191 , piece_id=C46MUQ5WXGG4F3WYNBQH3YBU4DXXXST2FHIKKPJ3SLFIK72UFIDA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.873452359 (577h4m8.744535394s) [noise=true , address=77.90.31.16:4736 , piece_id=7VY4YJ3GBZVSJEGKI5IZOWOGUJOR33EOLF6AUF67VY5GCVG7BFLA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:09.018634499 (577h4m8.889717534s) [noise=true , address=77.90.31.10:4519 , piece_id=M7CIKY6NNGADTISYWYNPP2YVJGGNOAXREIIGO4RHR54DZUW6YLUA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:09.018657852 (577h4m8.889740898s) [seg_pos={0 0}, upload=505882]: Segment reached optimal threshold of 13 pieces. Dec 29 16:33:09.018945279 (577h4m8.890028325s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.018970448 (577h4m8.890053493s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019101878 (577h4m8.890184923s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.019107689 (577h4m8.890190724s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019140251 (577h4m8.890223286s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.019171601 (577h4m8.890254646s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019194204 (577h4m8.890277249s) [seg_pos={0 0}, upload=505882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:33:09.019393022 (577h4m8.890476057s) [upload=505882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:33:09.019402090 (577h4m8.890485125s) [upload=505882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:33:09.318770071 (577h4m9.189853106s) [upload=505882]: Done uploading segments... Dec 29 16:33:09.318794417 (577h4m9.189877452s) [upload=505882]: Upload finished. err: <nil> Duration: 28.818184141s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..X] duration: 5.002260365s trace: https://jaeger.dev.storj.io/trace/2d30db106cc5c14d FAIL (2025-12-29 16:34:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002285804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 30.062033497s trace: https://jaeger.dev.storj.io/trace/25eb57d21bedad4d FAIL (2025-12-29 16:34:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, Piece ID: KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA) context deadline exceeded; piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ) context deadline exceeded; piecestore: (Node ID: 12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, Piece ID: MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA) context deadline exceeded ======LOGS====== 530948251s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660301397 (577h2m12.531384432s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660322538 (577h2m12.531405583s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660796670 (577h2m12.531879705s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661151005 (577h2m12.53223404s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661355344 (577h2m12.532438389s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661875545 (577h2m12.53295859s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662039136 (577h2m12.533122202s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662350129 (577h2m12.533433174s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662716055 (577h2m12.5337991s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662805526 (577h2m12.533888571s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.663913205 (577h2m12.53499624s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.664659345 (577h2m12.535742411s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.666957802 (577h2m12.538040847s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.668348230 (577h2m12.539431305s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:14.044161823 (577h2m13.915244868s) [upload=505855]: Next returned nil segment Dec 29 16:31:14.044175078 (577h2m13.915258113s) [upload=505855]: Waiting for error group managing segments... Dec 29 16:31:14.378564616 (577h2m14.249647661s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.513685195 (577h2m14.384768231s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.765759753 (577h2m14.636842788s) [noise=true , address=45.91.116.5:4316 , piece_id=BVVWSLENSRBMD5OBFJHPOBKDYHE55LLKPQ3MNLD4DCBPAKPUTGTA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.796869410 (577h2m14.667952445s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.805110142 (577h2m14.676193187s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.809656709 (577h2m14.680739744s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.828088656 (577h2m14.699171701s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.916522628 (577h2m14.787605663s) [noise=true , address=77.90.31.17:4795 , piece_id=QXXJ6VOBDRVCVRRMLSBQO7MB2DS5PUMDE7KPGOIJ77AJOJPSU7XQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.387028071 (577h2m15.258111106s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.424600454 (577h2m15.295683489s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.431096903 (577h2m15.302179939s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.454782875 (577h2m15.32586591s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.845291312 (577h2m15.716374357s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302138686 (577h2m16.173221731s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302163113 (577h2m16.173246148s) [seg_pos={0 0}, upload=505855]: Segment reached optimal threshold of 14 pieces. Dec 29 16:31:16.302359598 (577h2m16.173442663s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XmqDfLfXDjJ6Bo9PAYiVgshEBPnfQ1D9yC6CcvXpPRCWq4Wrs): protocol: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection Dec 29 16:31:16.302370097 (577h2m16.173453152s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302392490 (577h2m16.173475535s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection Dec 29 16:31:16.302401277 (577h2m16.173484322s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302574897 (577h2m16.173657952s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302582952 (577h2m16.173666018s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302723921 (577h2m16.173806956s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302731816 (577h2m16.173814851s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302750131 (577h2m16.173833176s) [seg_pos={0 0}, upload=505855]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:31:16.302842045 (577h2m16.17392509s) [upload=505855]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:31:16.302848428 (577h2m16.173931473s) [upload=505855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:31:16.589365678 (577h2m16.460448733s) [upload=505855]: Done uploading segments... Dec 29 16:31:16.589388892 (577h2m16.460471938s) [upload=505855]: Upload finished. err: <nil> Duration: 30.062053074s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X..] duration: 5.000731136s trace: https://jaeger.dev.storj.io/trace/169a5dccad7be0e5 FAIL (2025-12-29 16:34:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00075462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X....X.X......X.....X......X...XX.XXX....X...XXXXX.X..XX.] duration: 6.006477598s trace: https://jaeger.dev.storj.io/trace/24520496a51f996 FAIL (2025-12-29 16:33:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:33:05.637966222 (577h4m5.509049257s) [upload=505882]: Starting upload Dec 29 16:33:05.638021567 (577h4m5.509104602s) [upload=505882]: Done starting upload Dec 29 16:33:05.638067174 (577h4m5.50915021s) [upload=505882]: Uploading segments... Dec 29 16:33:05.638221718 (577h4m5.509304763s) [upload=505882]: Got next segment. Inline: false Dec 29 16:33:05.638253910 (577h4m5.509336945s) [upload=505882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:33:05.638258649 (577h4m5.509341684s) [upload=505882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:33:05.893573608 (577h4m5.764656643s) [seg_pos={0 0}, upload=505882]: Begin upload segment... Dec 29 16:33:05.893622982 (577h4m5.764706027s) [seg_pos={0 0}, upload=505882]: Done begin upload segment. Dec 29 16:33:05.893656255 (577h4m5.76473929s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.893669259 (577h4m5.764752294s) [noise=true , address=77.90.31.10:4519 , piece_id=M7CIKY6NNGADTISYWYNPP2YVJGGNOAXREIIGO4RHR54DZUW6YLUA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.894660668 (577h4m5.765743713s) [noise=true , address=145.79.227.5:4221 , piece_id=2SZWF3GDUPSQ5C6KXXCZYHNWCCMBY5UEASL4JNEW7LWHPDNPLGNQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.894728627 (577h4m5.765811672s) [noise=true , address=145.79.227.9:4366 , piece_id=ZI234HLBZR6H5XZYOB4ZX2GGPCE2Q5AQJXEQVIDIXO3BZ3RKZ6WA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.895460391 (577h4m5.766543436s) [noise=true , address=145.79.227.21:4475 , piece_id=GKUGCRAMQNBWNKD5TEDT7MXCW76Y5X6X264Z4WPTP7SCZGZPHAQA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.895854220 (577h4m5.766937255s) [noise=true , address=45.91.116.14:4191 , piece_id=C46MUQ5WXGG4F3WYNBQH3YBU4DXXXST2FHIKKPJ3SLFIK72UFIDA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.897017725 (577h4m5.76810077s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.897077018 (577h4m5.768160053s) [noise=true , address=77.90.31.19:4841 , piece_id=U2GDKURYAD6DBCDECSOXDKWKXKSN4LQ6MPHO3RO2H6KZAUJSCU5Q, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898015766 (577h4m5.769098801s) [noise=true , address=45.91.116.8:4047 , piece_id=MG4FDE5VWDEUIPNM4RZIXF3MVIJSESA7FUL42OMZTULOBV7VZB5A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898475862 (577h4m5.769558907s) [noise=true , address=145.79.227.6:4257 , piece_id=ZZJO76O6VOS7AYUSUJC3M422ZLNMEWQQ5FNRASAKCIH7QR6FKPBQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898971546 (577h4m5.770054581s) [noise=true , address=145.79.227.4:4186 , piece_id=O3XGN5IFZUWRAN6I4BEKOVPSJQJIETNNTNWR3FII3JLW46SIUCZQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899017404 (577h4m5.770100439s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYS6BRKSFUAY4PPEMBS3YKIH4RDTRXZ24QY5FNFJCUXVVCDA6HRQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899659075 (577h4m5.77074211s) [noise=true , address=145.79.227.7:4294 , piece_id=A4ACPZJB6KYLQQET6FYQPLX4SFYWNTOVC7PEWNDYRPIPYKRFAY5A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899723157 (577h4m5.770806202s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.900184796 (577h4m5.771267831s) [noise=true , address=77.90.31.16:4736 , piece_id=7VY4YJ3GBZVSJEGKI5IZOWOGUJOR33EOLF6AUF67VY5GCVG7BFLA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.900859971 (577h4m5.771943026s) [noise=true , address=145.79.227.11:4731 , piece_id=AD5NLKAMUYAD7CEUKLBZGPJWWRAE7PGR5QMFZ3HF3TXJSILZ6JHA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:07.586370754 (577h4m7.45745379s) [upload=505882]: Next returned nil segment Dec 29 16:33:07.586382908 (577h4m7.457465943s) [upload=505882]: Waiting for error group managing segments... Dec 29 16:33:07.871908942 (577h4m7.742991977s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYS6BRKSFUAY4PPEMBS3YKIH4RDTRXZ24QY5FNFJCUXVVCDA6HRQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.882984622 (577h4m7.754067657s) [noise=true , address=145.79.227.6:4257 , piece_id=ZZJO76O6VOS7AYUSUJC3M422ZLNMEWQQ5FNRASAKCIH7QR6FKPBQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.885295252 (577h4m7.756378297s) [noise=true , address=145.79.227.5:4221 , piece_id=2SZWF3GDUPSQ5C6KXXCZYHNWCCMBY5UEASL4JNEW7LWHPDNPLGNQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.902485234 (577h4m7.773568279s) [noise=true , address=145.79.227.11:4731 , piece_id=AD5NLKAMUYAD7CEUKLBZGPJWWRAE7PGR5QMFZ3HF3TXJSILZ6JHA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.904503718 (577h4m7.775586763s) [noise=true , address=145.79.227.4:4186 , piece_id=O3XGN5IFZUWRAN6I4BEKOVPSJQJIETNNTNWR3FII3JLW46SIUCZQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.906260594 (577h4m7.777343629s) [noise=true , address=145.79.227.9:4366 , piece_id=ZI234HLBZR6H5XZYOB4ZX2GGPCE2Q5AQJXEQVIDIXO3BZ3RKZ6WA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.909545378 (577h4m7.780628413s) [noise=true , address=145.79.227.21:4475 , piece_id=GKUGCRAMQNBWNKD5TEDT7MXCW76Y5X6X264Z4WPTP7SCZGZPHAQA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.987049296 (577h4m7.858132331s) [noise=true , address=145.79.227.7:4294 , piece_id=A4ACPZJB6KYLQQET6FYQPLX4SFYWNTOVC7PEWNDYRPIPYKRFAY5A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.080346628 (577h4m7.951429664s) [noise=true , address=77.90.31.19:4841 , piece_id=U2GDKURYAD6DBCDECSOXDKWKXKSN4LQ6MPHO3RO2H6KZAUJSCU5Q, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.108083083 (577h4m7.979166118s) [noise=true , address=45.91.116.8:4047 , piece_id=MG4FDE5VWDEUIPNM4RZIXF3MVIJSESA7FUL42OMZTULOBV7VZB5A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.143935421 (577h4m8.015018456s) [noise=true , address=45.91.116.14:4191 , piece_id=C46MUQ5WXGG4F3WYNBQH3YBU4DXXXST2FHIKKPJ3SLFIK72UFIDA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.873452359 (577h4m8.744535394s) [noise=true , address=77.90.31.16:4736 , piece_id=7VY4YJ3GBZVSJEGKI5IZOWOGUJOR33EOLF6AUF67VY5GCVG7BFLA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:09.018634499 (577h4m8.889717534s) [noise=true , address=77.90.31.10:4519 , piece_id=M7CIKY6NNGADTISYWYNPP2YVJGGNOAXREIIGO4RHR54DZUW6YLUA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:09.018657852 (577h4m8.889740898s) [seg_pos={0 0}, upload=505882]: Segment reached optimal threshold of 13 pieces. Dec 29 16:33:09.018945279 (577h4m8.890028325s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.018970448 (577h4m8.890053493s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019101878 (577h4m8.890184923s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.019107689 (577h4m8.890190724s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019140251 (577h4m8.890223286s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.019171601 (577h4m8.890254646s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019194204 (577h4m8.890277249s) [seg_pos={0 0}, upload=505882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:33:09.019393022 (577h4m8.890476057s) [upload=505882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:33:09.019402090 (577h4m8.890485125s) [upload=505882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:33:09.318770071 (577h4m9.189853106s) [upload=505882]: Done uploading segments... Dec 29 16:33:09.318794417 (577h4m9.189877452s) [upload=505882]: Upload finished. err: <nil> Duration: 6.006501033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X....X.X......X.....X......X...XX.XXX....X...XXXXX.X..XX] duration: 30.003370661s trace: https://jaeger.dev.storj.io/trace/24520496a51f996 FAIL (2025-12-29 16:33:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:33:05.637966222 (577h4m5.509049257s) [upload=505882]: Starting upload Dec 29 16:33:05.638021567 (577h4m5.509104602s) [upload=505882]: Done starting upload Dec 29 16:33:05.638067174 (577h4m5.50915021s) [upload=505882]: Uploading segments... Dec 29 16:33:05.638221718 (577h4m5.509304763s) [upload=505882]: Got next segment. Inline: false Dec 29 16:33:05.638253910 (577h4m5.509336945s) [upload=505882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:33:05.638258649 (577h4m5.509341684s) [upload=505882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:33:05.893573608 (577h4m5.764656643s) [seg_pos={0 0}, upload=505882]: Begin upload segment... Dec 29 16:33:05.893622982 (577h4m5.764706027s) [seg_pos={0 0}, upload=505882]: Done begin upload segment. Dec 29 16:33:05.893656255 (577h4m5.76473929s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.893669259 (577h4m5.764752294s) [noise=true , address=77.90.31.10:4519 , piece_id=M7CIKY6NNGADTISYWYNPP2YVJGGNOAXREIIGO4RHR54DZUW6YLUA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.894660668 (577h4m5.765743713s) [noise=true , address=145.79.227.5:4221 , piece_id=2SZWF3GDUPSQ5C6KXXCZYHNWCCMBY5UEASL4JNEW7LWHPDNPLGNQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.894728627 (577h4m5.765811672s) [noise=true , address=145.79.227.9:4366 , piece_id=ZI234HLBZR6H5XZYOB4ZX2GGPCE2Q5AQJXEQVIDIXO3BZ3RKZ6WA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.895460391 (577h4m5.766543436s) [noise=true , address=145.79.227.21:4475 , piece_id=GKUGCRAMQNBWNKD5TEDT7MXCW76Y5X6X264Z4WPTP7SCZGZPHAQA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.895854220 (577h4m5.766937255s) [noise=true , address=45.91.116.14:4191 , piece_id=C46MUQ5WXGG4F3WYNBQH3YBU4DXXXST2FHIKKPJ3SLFIK72UFIDA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.897017725 (577h4m5.76810077s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.897077018 (577h4m5.768160053s) [noise=true , address=77.90.31.19:4841 , piece_id=U2GDKURYAD6DBCDECSOXDKWKXKSN4LQ6MPHO3RO2H6KZAUJSCU5Q, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898015766 (577h4m5.769098801s) [noise=true , address=45.91.116.8:4047 , piece_id=MG4FDE5VWDEUIPNM4RZIXF3MVIJSESA7FUL42OMZTULOBV7VZB5A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898475862 (577h4m5.769558907s) [noise=true , address=145.79.227.6:4257 , piece_id=ZZJO76O6VOS7AYUSUJC3M422ZLNMEWQQ5FNRASAKCIH7QR6FKPBQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.898971546 (577h4m5.770054581s) [noise=true , address=145.79.227.4:4186 , piece_id=O3XGN5IFZUWRAN6I4BEKOVPSJQJIETNNTNWR3FII3JLW46SIUCZQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899017404 (577h4m5.770100439s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYS6BRKSFUAY4PPEMBS3YKIH4RDTRXZ24QY5FNFJCUXVVCDA6HRQ, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899659075 (577h4m5.77074211s) [noise=true , address=145.79.227.7:4294 , piece_id=A4ACPZJB6KYLQQET6FYQPLX4SFYWNTOVC7PEWNDYRPIPYKRFAY5A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.899723157 (577h4m5.770806202s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.900184796 (577h4m5.771267831s) [noise=true , address=77.90.31.16:4736 , piece_id=7VY4YJ3GBZVSJEGKI5IZOWOGUJOR33EOLF6AUF67VY5GCVG7BFLA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:05.900859971 (577h4m5.771943026s) [noise=true , address=145.79.227.11:4731 , piece_id=AD5NLKAMUYAD7CEUKLBZGPJWWRAE7PGR5QMFZ3HF3TXJSILZ6JHA, seg_pos={0 0}, upload=505882]: Uploading piece... Dec 29 16:33:07.586370754 (577h4m7.45745379s) [upload=505882]: Next returned nil segment Dec 29 16:33:07.586382908 (577h4m7.457465943s) [upload=505882]: Waiting for error group managing segments... Dec 29 16:33:07.871908942 (577h4m7.742991977s) [noise=true , address=145.79.227.51:4410 , piece_id=ZYS6BRKSFUAY4PPEMBS3YKIH4RDTRXZ24QY5FNFJCUXVVCDA6HRQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.882984622 (577h4m7.754067657s) [noise=true , address=145.79.227.6:4257 , piece_id=ZZJO76O6VOS7AYUSUJC3M422ZLNMEWQQ5FNRASAKCIH7QR6FKPBQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.885295252 (577h4m7.756378297s) [noise=true , address=145.79.227.5:4221 , piece_id=2SZWF3GDUPSQ5C6KXXCZYHNWCCMBY5UEASL4JNEW7LWHPDNPLGNQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.902485234 (577h4m7.773568279s) [noise=true , address=145.79.227.11:4731 , piece_id=AD5NLKAMUYAD7CEUKLBZGPJWWRAE7PGR5QMFZ3HF3TXJSILZ6JHA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.904503718 (577h4m7.775586763s) [noise=true , address=145.79.227.4:4186 , piece_id=O3XGN5IFZUWRAN6I4BEKOVPSJQJIETNNTNWR3FII3JLW46SIUCZQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.906260594 (577h4m7.777343629s) [noise=true , address=145.79.227.9:4366 , piece_id=ZI234HLBZR6H5XZYOB4ZX2GGPCE2Q5AQJXEQVIDIXO3BZ3RKZ6WA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.909545378 (577h4m7.780628413s) [noise=true , address=145.79.227.21:4475 , piece_id=GKUGCRAMQNBWNKD5TEDT7MXCW76Y5X6X264Z4WPTP7SCZGZPHAQA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:07.987049296 (577h4m7.858132331s) [noise=true , address=145.79.227.7:4294 , piece_id=A4ACPZJB6KYLQQET6FYQPLX4SFYWNTOVC7PEWNDYRPIPYKRFAY5A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.080346628 (577h4m7.951429664s) [noise=true , address=77.90.31.19:4841 , piece_id=U2GDKURYAD6DBCDECSOXDKWKXKSN4LQ6MPHO3RO2H6KZAUJSCU5Q, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.108083083 (577h4m7.979166118s) [noise=true , address=45.91.116.8:4047 , piece_id=MG4FDE5VWDEUIPNM4RZIXF3MVIJSESA7FUL42OMZTULOBV7VZB5A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.143935421 (577h4m8.015018456s) [noise=true , address=45.91.116.14:4191 , piece_id=C46MUQ5WXGG4F3WYNBQH3YBU4DXXXST2FHIKKPJ3SLFIK72UFIDA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:08.873452359 (577h4m8.744535394s) [noise=true , address=77.90.31.16:4736 , piece_id=7VY4YJ3GBZVSJEGKI5IZOWOGUJOR33EOLF6AUF67VY5GCVG7BFLA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:09.018634499 (577h4m8.889717534s) [noise=true , address=77.90.31.10:4519 , piece_id=M7CIKY6NNGADTISYWYNPP2YVJGGNOAXREIIGO4RHR54DZUW6YLUA, seg_pos={0 0}, upload=505882]: Done uploading piece. err: <nil> Dec 29 16:33:09.018657852 (577h4m8.889740898s) [seg_pos={0 0}, upload=505882]: Segment reached optimal threshold of 13 pieces. Dec 29 16:33:09.018945279 (577h4m8.890028325s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.018970448 (577h4m8.890053493s) [noise=true , address=145.79.227.34:4905 , piece_id=HYLU6NFVWGW47HU46SWXCHWKRZPBD45WEKVKY2727HQC6XLEYZHQ, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019101878 (577h4m8.890184923s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.019107689 (577h4m8.890190724s) [noise=true , address=45.91.116.13:4154 , piece_id=RVQBWJZL2Q3QY5DHRDOLBWXSCVZCHGYPDPP6G5X5U7RFQPHFW53A, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019140251 (577h4m8.890223286s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 16:33:09.019171601 (577h4m8.890254646s) [noise=true , address=45.91.116.4:4937 , piece_id=6KYLU6N42P4D4RYRIIEDWCMWIWEANMFXIUWWCSDQ5MWJYJZXW7NQ, seg_pos={0 0}, upload=505882]: Long tail cancellation due to optimal threshold Dec 29 16:33:09.019194204 (577h4m8.890277249s) [seg_pos={0 0}, upload=505882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:33:09.019393022 (577h4m8.890476057s) [upload=505882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:33:09.019402090 (577h4m8.890485125s) [upload=505882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:33:09.318770071 (577h4m9.189853106s) [upload=505882]: Done uploading segments... Dec 29 16:33:09.318794417 (577h4m9.189877452s) [upload=505882]: Upload finished. err: <nil> Duration: 30.003393015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 42.642788749s trace: https://jaeger.dev.storj.io/trace/25eb57d21bedad4d FAIL (2025-12-29 16:33:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, Piece ID: KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA) context deadline exceeded; piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ) context deadline exceeded; piecestore: (Node ID: 12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, Piece ID: MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA) context deadline exceeded ======LOGS====== 530948251s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660301397 (577h2m12.531384432s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660322538 (577h2m12.531405583s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660796670 (577h2m12.531879705s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661151005 (577h2m12.53223404s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661355344 (577h2m12.532438389s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661875545 (577h2m12.53295859s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662039136 (577h2m12.533122202s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662350129 (577h2m12.533433174s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662716055 (577h2m12.5337991s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662805526 (577h2m12.533888571s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.663913205 (577h2m12.53499624s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.664659345 (577h2m12.535742411s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.666957802 (577h2m12.538040847s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.668348230 (577h2m12.539431305s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:14.044161823 (577h2m13.915244868s) [upload=505855]: Next returned nil segment Dec 29 16:31:14.044175078 (577h2m13.915258113s) [upload=505855]: Waiting for error group managing segments... Dec 29 16:31:14.378564616 (577h2m14.249647661s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.513685195 (577h2m14.384768231s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.765759753 (577h2m14.636842788s) [noise=true , address=45.91.116.5:4316 , piece_id=BVVWSLENSRBMD5OBFJHPOBKDYHE55LLKPQ3MNLD4DCBPAKPUTGTA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.796869410 (577h2m14.667952445s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.805110142 (577h2m14.676193187s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.809656709 (577h2m14.680739744s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.828088656 (577h2m14.699171701s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.916522628 (577h2m14.787605663s) [noise=true , address=77.90.31.17:4795 , piece_id=QXXJ6VOBDRVCVRRMLSBQO7MB2DS5PUMDE7KPGOIJ77AJOJPSU7XQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.387028071 (577h2m15.258111106s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.424600454 (577h2m15.295683489s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.431096903 (577h2m15.302179939s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.454782875 (577h2m15.32586591s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.845291312 (577h2m15.716374357s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302138686 (577h2m16.173221731s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302163113 (577h2m16.173246148s) [seg_pos={0 0}, upload=505855]: Segment reached optimal threshold of 14 pieces. Dec 29 16:31:16.302359598 (577h2m16.173442663s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XmqDfLfXDjJ6Bo9PAYiVgshEBPnfQ1D9yC6CcvXpPRCWq4Wrs): protocol: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection Dec 29 16:31:16.302370097 (577h2m16.173453152s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302392490 (577h2m16.173475535s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection Dec 29 16:31:16.302401277 (577h2m16.173484322s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302574897 (577h2m16.173657952s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302582952 (577h2m16.173666018s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302723921 (577h2m16.173806956s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302731816 (577h2m16.173814851s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302750131 (577h2m16.173833176s) [seg_pos={0 0}, upload=505855]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:31:16.302842045 (577h2m16.17392509s) [upload=505855]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:31:16.302848428 (577h2m16.173931473s) [upload=505855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:31:16.589365678 (577h2m16.460448733s) [upload=505855]: Done uploading segments... Dec 29 16:31:16.589388892 (577h2m16.460471938s) [upload=505855]: Upload finished. err: <nil> Duration: 42.642803236s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X.] duration: 5.000747295s trace: https://jaeger.dev.storj.io/trace/169a5dccad7be0e5 FAIL (2025-12-29 16:33:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000774908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 37.389004501s trace: https://jaeger.dev.storj.io/trace/25eb57d21bedad4d FAIL (2025-12-29 16:32:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, Piece ID: KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA) context deadline exceeded; piecestore: (Node ID: 12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, Piece ID: QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ) context deadline exceeded; piecestore: (Node ID: 12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, Piece ID: MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA) context deadline exceeded ======LOGS====== 530948251s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660301397 (577h2m12.531384432s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660322538 (577h2m12.531405583s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.660796670 (577h2m12.531879705s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661151005 (577h2m12.53223404s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661355344 (577h2m12.532438389s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.661875545 (577h2m12.53295859s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662039136 (577h2m12.533122202s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662350129 (577h2m12.533433174s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662716055 (577h2m12.5337991s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.662805526 (577h2m12.533888571s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.663913205 (577h2m12.53499624s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.664659345 (577h2m12.535742411s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.666957802 (577h2m12.538040847s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:12.668348230 (577h2m12.539431305s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Uploading piece... Dec 29 16:31:14.044161823 (577h2m13.915244868s) [upload=505855]: Next returned nil segment Dec 29 16:31:14.044175078 (577h2m13.915258113s) [upload=505855]: Waiting for error group managing segments... Dec 29 16:31:14.378564616 (577h2m14.249647661s) [noise=true , address=64.49.13.138:4484 , piece_id=LYLYLZVHLT6EMZHVB7FWHR2VZPHVYD4FTNCXWGIY4COI6K7JI3ZA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.513685195 (577h2m14.384768231s) [noise=true , address=145.79.227.48:4315 , piece_id=LO7JE4O374R4J6APPJDSHYYBE6SLJSKZSP2IBW3Q25PV4GGRQE2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.765759753 (577h2m14.636842788s) [noise=true , address=45.91.116.5:4316 , piece_id=BVVWSLENSRBMD5OBFJHPOBKDYHE55LLKPQ3MNLD4DCBPAKPUTGTA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.796869410 (577h2m14.667952445s) [noise=true , address=77.90.31.16:4748 , piece_id=RTTRB26LNSASQF6SAXXXFHTYA5JSX6SOUFZMP2PJEQEQAU26GGFQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.805110142 (577h2m14.676193187s) [noise=true , address=45.91.116.8:4061 , piece_id=P3TLNRR5HQLCCMB5UVXFSMEVSRIIAG43WXYOZRBFAK5G3UCYJ2LQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.809656709 (577h2m14.680739744s) [noise=true , address=77.90.31.11:4579 , piece_id=MHQMFSAE5IZLW7L6SNJOOZNQGRES54BUYHOWCVUSR5HSCWUIIXYQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.828088656 (577h2m14.699171701s) [noise=true , address=77.90.31.13:4642 , piece_id=KER55IHIMR4FYX7PFGZEY6ODWVGDM57KFNNOZEZR22AXEGEEYENA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:14.916522628 (577h2m14.787605663s) [noise=true , address=77.90.31.17:4795 , piece_id=QXXJ6VOBDRVCVRRMLSBQO7MB2DS5PUMDE7KPGOIJ77AJOJPSU7XQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.387028071 (577h2m15.258111106s) [noise=true , address=77.90.31.12:4619 , piece_id=MZHPGSQPAAWGXZNCQI32455B2AUO6SZYNH6V2BLOAPQW5ZPUQHHA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.424600454 (577h2m15.295683489s) [noise=true , address=145.79.227.40:4069 , piece_id=3ZG4ESMUIBTAXW26PGREEQIKUUJHWXTLQUIAZEM5S7FY6DNY43MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.431096903 (577h2m15.302179939s) [noise=true , address=145.79.227.44:4225 , piece_id=YAGPTXDNKORDDGC2H5W6L4JQFIF6JJDIQRNKC57DCVNNYBSHCMFA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.454782875 (577h2m15.32586591s) [noise=true , address=145.79.227.43:4169 , piece_id=BFPI72K2I4C2O5PP7Y5SMBN7JAAWL2FQX472HBROBAABWQK6CONQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:15.845291312 (577h2m15.716374357s) [noise=true , address=77.90.31.15:4717 , piece_id=QWGNXY5J6FDVKFT3TK4SYNDIL3IDSELB2SMUCDXX7XOEZX3HKXDQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302138686 (577h2m16.173221731s) [noise=true , address=45.91.116.7:4040 , piece_id=42M4M6YE3SFC3D3VUU4UNK3M46V7LHZJZ7IB6PW6MNOGOMXAGQ2A, seg_pos={0 0}, upload=505855]: Done uploading piece. err: <nil> Dec 29 16:31:16.302163113 (577h2m16.173246148s) [seg_pos={0 0}, upload=505855]: Segment reached optimal threshold of 14 pieces. Dec 29 16:31:16.302359598 (577h2m16.173442663s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XmqDfLfXDjJ6Bo9PAYiVgshEBPnfQ1D9yC6CcvXpPRCWq4Wrs): protocol: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52416->64.49.13.27:4958: use of closed network connection Dec 29 16:31:16.302370097 (577h2m16.173453152s) [noise=true , address=64.49.13.27:4958 , piece_id=FHUY5375HCJIISRKJVAIDV44GORQP6NC2KMOLXT5YP34FZAJC3MA, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302392490 (577h2m16.173475535s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37572->64.49.13.169:4705: use of closed network connection Dec 29 16:31:16.302401277 (577h2m16.173484322s) [noise=true , address=64.49.13.169:4705 , piece_id=3NRLU6IP7AZIW4F7D3TAWSZOXPUUUNTOVJVCAR3G4BA7UU2SFDEQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302574897 (577h2m16.173657952s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302582952 (577h2m16.173666018s) [noise=true , address=145.79.227.35:4511 , piece_id=HIURTMGQK2WF4776XCL3UQFUNFXKT6SAVLAJ5Y3FGZYPSSHWT3UQ, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302723921 (577h2m16.173806956s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:16.302731816 (577h2m16.173814851s) [noise=true , address=77.90.31.11:4562 , piece_id=5QXWD7YPMEEIBLDO7JNKS5JUMIONRXKL4IE2QCYLUQY6ZLQF3N2Q, seg_pos={0 0}, upload=505855]: Long tail cancellation due to optimal threshold Dec 29 16:31:16.302750131 (577h2m16.173833176s) [seg_pos={0 0}, upload=505855]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 16:31:16.302842045 (577h2m16.17392509s) [upload=505855]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:31:16.302848428 (577h2m16.173931473s) [upload=505855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:31:16.589365678 (577h2m16.460448733s) [upload=505855]: Done uploading segments... Dec 29 16:31:16.589388892 (577h2m16.460471938s) [upload=505855]: Upload finished. err: <nil> Duration: 37.389020532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....X....X.X......X.....X......X...XX.XXX....X...XXXXX.X..X] duration: 30.002894442s trace: https://jaeger.dev.storj.io/trace/39606b308b6db86d FAIL (2025-12-29 16:32:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 16:31:52.717898365 (577h2m52.5889814s) [upload=505864]: Starting upload Dec 29 16:31:52.717955243 (577h2m52.589038288s) [upload=505864]: Done starting upload Dec 29 16:31:52.718100279 (577h2m52.589183324s) [upload=505864]: Uploading segments... Dec 29 16:31:52.718218875 (577h2m52.58930192s) [upload=505864]: Got next segment. Inline: false Dec 29 16:31:52.718261656 (577h2m52.589344691s) [upload=505864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 16:31:52.718268970 (577h2m52.589352015s) [upload=505864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 16:31:52.971879202 (577h2m52.842962247s) [seg_pos={0 0}, upload=505864]: Begin upload segment... Dec 29 16:31:52.971936751 (577h2m52.843019786s) [seg_pos={0 0}, upload=505864]: Done begin upload segment. Dec 29 16:31:52.971980535 (577h2m52.84306358s) [noise=true , address=45.91.116.13:4153 , piece_id=4HDJFSYEM7CL3DM5QKQ6YHFCFOJHV5FJIHPA4MIUTFMOS4UPIH2A, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.972003739 (577h2m52.843086774s) [noise=true , address=145.79.227.42:4123 , piece_id=I5M2GYJNYAXM7EZSACPWXYGTD5LHKTCDFLPR2QNUU65AJLBNJBAA, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.972471179 (577h2m52.843554215s) [noise=true , address=145.79.227.16:4909 , piece_id=QMBD7KLVQLVCBQU4SVH7EJLTJDAW7JM2LWCYXMZZV7MBIQKSOQFA, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.972484635 (577h2m52.84356768s) [noise=true , address=77.90.31.11:4554 , piece_id=2UDYJJ5OBSNCATZW6DPAEELDC7BZ6TC4243OYBTMG7OBWV3CUVEQ, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.972871752 (577h2m52.843954787s) [noise=true , address=145.79.227.43:4158 , piece_id=IVTX4XLSJCMF6EROZDBTXNF2H5IOEKMVLMA7RZWQO2CYVGGAOYBQ, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.972893173 (577h2m52.843976208s) [noise=true , address=145.79.227.21:4475 , piece_id=FPB5WLSCMWJA2YLMX2YUPFPCDVMDTHRNYWTQ4QZKZGERNN6WQJFA, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.973299847 (577h2m52.844382892s) [noise=true , address=145.79.227.46:4602 , piece_id=BNB7ASIWKT7Y5CXXCPNYHRG46KCRLZZ25JGR6YC7VGHGQCWXR5AA, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.973334723 (577h2m52.844417768s) [noise=true , address=145.79.227.31:4799 , piece_id=2JFKDBRODFSUDFWNTTFQCEDENLJQGWGJOM4745K3VLAAN2Q4AEKQ, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.973640996 (577h2m52.844724031s) [noise=true , address=145.79.227.9:4366 , piece_id=PN2YSZ73VCTZYO7VMWWHWKYRWHIKDN2PYPSEF4LY5RBIV3R75Y4A, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.973713564 (577h2m52.84479661s) [noise=true , address=145.79.227.44:4193 , piece_id=GOD7CFEV3BAT3WMAXUIAJCJAAZV5HKRSD7HTZNWNE2GDW7A22UUQ, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.973993437 (577h2m52.845076472s) [noise=true , address=45.91.116.14:4192 , piece_id=4NAABT3AFEELKNHIPCRCLNFEH4URQ5N2RNXQE4U25WR4XC2F2TMA, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.974635060 (577h2m52.845718095s) [noise=true , address=145.79.227.47:4265 , piece_id=UURJRNQ3Z6QDE4Q2NNG3KWMQKWUR3FEHZHWFMJYKDFQHT66FE6FQ, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.975716960 (577h2m52.846799995s) [noise=true , address=145.79.227.13:4803 , piece_id=77CXINIPTJI5IEMAKUQ6ZPIU756JWYDREFRAFISOB4M2BPI3EVHA, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.976357689 (577h2m52.847440734s) [noise=true , address=45.91.116.7:4012 , piece_id=7H7HD47WFUXRADJZF3PMS6PPNXJELXWRJVNHSIQBOD5TZZXPPFKQ, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.976778621 (577h2m52.847861666s) [noise=true , address=45.91.116.5:4300 , piece_id=QTYWORRNTQXM4MXGFN2P2LFZNACAG2CD657WSP63QY4BHZEXVSMA, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:52.978644043 (577h2m52.849727088s) [noise=true , address=145.79.227.30:4761 , piece_id=ZW3PWQATRSNDFC4OAYGYLDJNOWMKIMM7NOAPV4ZMJUFKWWRXPUMA, seg_pos={0 0}, upload=505864]: Uploading piece... Dec 29 16:31:54.572618488 (577h2m54.443701523s) [upload=505864]: Next returned nil segment Dec 29 16:31:54.572636904 (577h2m54.443719939s) [upload=505864]: Waiting for error group managing segments... Dec 29 16:31:54.955480956 (577h2m54.826563991s) [noise=true , address=145.79.227.13:4803 , piece_id=77CXINIPTJI5IEMAKUQ6ZPIU756JWYDREFRAFISOB4M2BPI3EVHA, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:54.961311728 (577h2m54.832394763s) [noise=true , address=145.79.227.42:4123 , piece_id=I5M2GYJNYAXM7EZSACPWXYGTD5LHKTCDFLPR2QNUU65AJLBNJBAA, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:54.963318379 (577h2m54.834401414s) [noise=true , address=145.79.227.21:4475 , piece_id=FPB5WLSCMWJA2YLMX2YUPFPCDVMDTHRNYWTQ4QZKZGERNN6WQJFA, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:54.965554688 (577h2m54.836637723s) [noise=true , address=145.79.227.47:4265 , piece_id=UURJRNQ3Z6QDE4Q2NNG3KWMQKWUR3FEHZHWFMJYKDFQHT66FE6FQ, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:54.974483249 (577h2m54.845566284s) [noise=true , address=145.79.227.16:4909 , piece_id=QMBD7KLVQLVCBQU4SVH7EJLTJDAW7JM2LWCYXMZZV7MBIQKSOQFA, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:54.979504350 (577h2m54.850587395s) [noise=true , address=145.79.227.31:4799 , piece_id=2JFKDBRODFSUDFWNTTFQCEDENLJQGWGJOM4745K3VLAAN2Q4AEKQ, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:54.980211215 (577h2m54.85129425s) [noise=true , address=145.79.227.46:4602 , piece_id=BNB7ASIWKT7Y5CXXCPNYHRG46KCRLZZ25JGR6YC7VGHGQCWXR5AA, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:55.001697819 (577h2m54.872780874s) [noise=true , address=145.79.227.9:4366 , piece_id=PN2YSZ73VCTZYO7VMWWHWKYRWHIKDN2PYPSEF4LY5RBIV3R75Y4A, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:55.230092577 (577h2m55.101175612s) [noise=true , address=45.91.116.14:4192 , piece_id=4NAABT3AFEELKNHIPCRCLNFEH4URQ5N2RNXQE4U25WR4XC2F2TMA, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:55.268752831 (577h2m55.139835877s) [noise=true , address=77.90.31.11:4554 , piece_id=2UDYJJ5OBSNCATZW6DPAEELDC7BZ6TC4243OYBTMG7OBWV3CUVEQ, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:55.300582560 (577h2m55.171665605s) [noise=true , address=45.91.116.7:4012 , piece_id=7H7HD47WFUXRADJZF3PMS6PPNXJELXWRJVNHSIQBOD5TZZXPPFKQ, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:55.303295687 (577h2m55.174378722s) [noise=true , address=45.91.116.5:4300 , piece_id=QTYWORRNTQXM4MXGFN2P2LFZNACAG2CD657WSP63QY4BHZEXVSMA, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:55.704429025 (577h2m55.57551207s) [noise=true , address=45.91.116.13:4153 , piece_id=4HDJFSYEM7CL3DM5QKQ6YHFCFOJHV5FJIHPA4MIUTFMOS4UPIH2A, seg_pos={0 0}, upload=505864]: Done uploading piece. err: <nil> Dec 29 16:31:55.704451276 (577h2m55.575534321s) [seg_pos={0 0}, upload=505864]: Segment reached optimal threshold of 13 pieces. Dec 29 16:31:55.704663941 (577h2m55.575746976s) [noise=true , address=145.79.227.30:4761 , piece_id=ZW3PWQATRSNDFC4OAYGYLDJNOWMKIMM7NOAPV4ZMJUFKWWRXPUMA, seg_pos={0 0}, upload=505864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:55.704671657 (577h2m55.575754692s) [noise=true , address=145.79.227.30:4761 , piece_id=ZW3PWQATRSNDFC4OAYGYLDJNOWMKIMM7NOAPV4ZMJUFKWWRXPUMA, seg_pos={0 0}, upload=505864]: Long tail cancellation due to optimal threshold Dec 29 16:31:55.704799199 (577h2m55.575882234s) [noise=true , address=145.79.227.44:4193 , piece_id=GOD7CFEV3BAT3WMAXUIAJCJAAZV5HKRSD7HTZNWNE2GDW7A22UUQ, seg_pos={0 0}, upload=505864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:55.704806293 (577h2m55.575889378s) [noise=true , address=145.79.227.44:4193 , piece_id=GOD7CFEV3BAT3WMAXUIAJCJAAZV5HKRSD7HTZNWNE2GDW7A22UUQ, seg_pos={0 0}, upload=505864]: Long tail cancellation due to optimal threshold Dec 29 16:31:55.705143996 (577h2m55.576227041s) [noise=true , address=145.79.227.43:4158 , piece_id=IVTX4XLSJCMF6EROZDBTXNF2H5IOEKMVLMA7RZWQO2CYVGGAOYBQ, seg_pos={0 0}, upload=505864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 16:31:55.705178000 (577h2m55.576261045s) [noise=true , address=145.79.227.43:4158 , piece_id=IVTX4XLSJCMF6EROZDBTXNF2H5IOEKMVLMA7RZWQO2CYVGGAOYBQ, seg_pos={0 0}, upload=505864]: Long tail cancellation due to optimal threshold Dec 29 16:31:55.705203820 (577h2m55.576286855s) [seg_pos={0 0}, upload=505864]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 16:31:55.705339859 (577h2m55.576422894s) [upload=505864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 16:31:55.705345369 (577h2m55.576428404s) [upload=505864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 16:31:56.007511649 (577h2m55.878594694s) [upload=505864]: Done uploading segments... Dec 29 16:31:56.007535314 (577h2m55.878618349s) [upload=505864]: Upload finished. err: <nil> Duration: 30.002917296s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXX.X.X.XX..XXXX.XXX.XXXX.....X] duration: 5.001209527s trace: https://jaeger.dev.storj.io/trace/169a5dccad7be0e5 FAIL (2025-12-29 16:32:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001230907s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)