Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...XXX..X.X.....X........XX....X..XX............X..X.X..XXX] duration: 30.002696062s trace: https://jaeger.dev.storj.io/trace/4e2f8b596c8cf57c FAIL (2025-12-31 09:24:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: AHTIUGWB42LTI37NI6TEWL2ZQ7A5354IA2V25WW7DIY3FY3I5XIQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: IXNZH7SCVEUL4IIX5GXMHACPMYWOUC4WCYXAN4F6KVLRHBZ6EFEA) context deadline exceeded ======LOGS====== 31 09:23:31.599935720 (617h54m31.471018765s) [upload=541490]: Starting upload Dec 31 09:23:31.600022084 (617h54m31.471105139s) [upload=541490]: Done starting upload Dec 31 09:23:31.600252002 (617h54m31.471335047s) [upload=541490]: Uploading segments... Dec 31 09:23:31.600457694 (617h54m31.471540739s) [upload=541490]: Got next segment. Inline: false Dec 31 09:23:31.600493432 (617h54m31.471576467s) [upload=541490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:23:31.600499173 (617h54m31.471582208s) [upload=541490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:23:31.843909651 (617h54m31.714992696s) [seg_pos={0 0}, upload=541490]: Begin upload segment... Dec 31 09:23:31.844212378 (617h54m31.715295413s) [seg_pos={0 0}, upload=541490]: Done begin upload segment. Dec 31 09:23:31.844244569 (617h54m31.715327614s) [noise=true , address=145.79.227.16:4912 , piece_id=L7COJ5QFU6ZJFX536B6J3USZRQZDDSTBDFLUUWLUI3UI2ZHH2VXQ, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.844256962 (617h54m31.715340007s) [noise=true , address=45.91.116.10:4082 , piece_id=IXNZH7SCVEUL4IIX5GXMHACPMYWOUC4WCYXAN4F6KVLRHBZ6EFEA, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.844435432 (617h54m31.715518467s) [noise=true , address=145.79.227.7:4295 , piece_id=7IB6QW7OVNWVLEKU5Y6UC7W2FDS5UMDRCKWLRUN5RQUVAHSJ5DIA, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.844856924 (617h54m31.715939969s) [noise=true , address=45.91.116.4:4940 , piece_id=DGDAW2WEIMBZJVJJKB2Z63EXSUZ67BYKDBKJ3QETAXM6VFBAQD7Q, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.845307883 (617h54m31.716390928s) [noise=true , address=145.79.227.41:4085 , piece_id=KSBBPWMVRKQGCOSGGTCFGA445HYQ36CD33AGN6YJ6ML45WG7JQAA, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.845927112 (617h54m31.717010157s) [noise=true , address=45.91.116.7:4012 , piece_id=G7Z63HRUBGSN6CK4ZNWJ6QGL2XAS26QHHYAWT6WYG7NE3Q3M57PA, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.846384874 (617h54m31.717467919s) [noise=true , address=145.79.227.5:4221 , piece_id=2JOMLXFLODQJ6AP4LRCUUCM5CQXWG35TWVDDV2AJ3OXTFZHQPW2Q, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.848124466 (617h54m31.719207501s) [noise=true , address=45.91.116.13:4156 , piece_id=2AX67JV6G5QBHSNXRXV2CVWSXRGRBI6SSQ52CWYI7HIHT2DG6R4Q, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.848646611 (617h54m31.719729656s) [noise=true , address=45.91.116.8:4047 , piece_id=YJFG3OZJYACXVQZUDJCQB5RES3HCHOHI7DUTO6WI75KJ3QDOC7VA, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.850752181 (617h54m31.721835226s) [noise=true , address=45.91.116.9:4264 , piece_id=AHTIUGWB42LTI37NI6TEWL2ZQ7A5354IA2V25WW7DIY3FY3I5XIQ, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.853407757 (617h54m31.724490802s) [noise=true , address=145.79.227.14:4840 , piece_id=BAYFQHGSENA4IZ5CMCVMPTJRXT72KSSVKK3267ZL5NPCYOOSZCMQ, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.856362493 (617h54m31.727445538s) [noise=true , address=145.79.227.36:4517 , piece_id=C6SGLKPOIA6RPBBYPGGKGC7Y53YTDH7NGG6FRDIYTRDRLNYMER3A, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.857720740 (617h54m31.728803775s) [noise=true , address=145.79.227.44:4193 , piece_id=NF7UFT2Y4V57XMXN42KC2JIZBZT47MYHU7RDB7IQDG4CB2MKN7OQ, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.860670205 (617h54m31.73175325s) [noise=true , address=145.79.227.13:4802 , piece_id=DLLDRWVS34OIILYIY7XVOVNXLX5PQKANY23AS6QT73GGLPOWDSOA, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.861920897 (617h54m31.733003932s) [noise=true , address=145.79.227.4:4185 , piece_id=OJSY62I2DY2Z5JIA66GGQW7LBDECEJJCKAHSSG476KYTT77SLBGQ, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:31.864562377 (617h54m31.735645412s) [noise=true , address=145.79.227.2:4114 , piece_id=2KVMGSC752EDLDLATMNAH4567HDDVPF4INJJL6NNJIVSTY2TYJUA, seg_pos={0 0}, upload=541490]: Uploading piece... Dec 31 09:23:33.548105753 (617h54m33.419188798s) [upload=541490]: Next returned nil segment Dec 31 09:23:33.548118287 (617h54m33.419201322s) [upload=541490]: Waiting for error group managing segments... Dec 31 09:23:33.819434472 (617h54m33.690517517s) [noise=true , address=145.79.227.41:4085 , piece_id=KSBBPWMVRKQGCOSGGTCFGA445HYQ36CD33AGN6YJ6ML45WG7JQAA, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:33.834287223 (617h54m33.705370268s) [noise=true , address=145.79.227.5:4221 , piece_id=2JOMLXFLODQJ6AP4LRCUUCM5CQXWG35TWVDDV2AJ3OXTFZHQPW2Q, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:33.843480980 (617h54m33.714564025s) [noise=true , address=145.79.227.14:4840 , piece_id=BAYFQHGSENA4IZ5CMCVMPTJRXT72KSSVKK3267ZL5NPCYOOSZCMQ, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:33.849996256 (617h54m33.721079301s) [noise=true , address=145.79.227.2:4114 , piece_id=2KVMGSC752EDLDLATMNAH4567HDDVPF4INJJL6NNJIVSTY2TYJUA, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:33.851003003 (617h54m33.722086048s) [noise=true , address=145.79.227.36:4517 , piece_id=C6SGLKPOIA6RPBBYPGGKGC7Y53YTDH7NGG6FRDIYTRDRLNYMER3A, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:33.868509638 (617h54m33.739592673s) [noise=true , address=145.79.227.4:4185 , piece_id=OJSY62I2DY2Z5JIA66GGQW7LBDECEJJCKAHSSG476KYTT77SLBGQ, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:33.905200953 (617h54m33.776283988s) [noise=true , address=145.79.227.16:4912 , piece_id=L7COJ5QFU6ZJFX536B6J3USZRQZDDSTBDFLUUWLUI3UI2ZHH2VXQ, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:33.919463371 (617h54m33.790546406s) [noise=true , address=145.79.227.7:4295 , piece_id=7IB6QW7OVNWVLEKU5Y6UC7W2FDS5UMDRCKWLRUN5RQUVAHSJ5DIA, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:33.932008408 (617h54m33.803091443s) [noise=true , address=145.79.227.13:4802 , piece_id=DLLDRWVS34OIILYIY7XVOVNXLX5PQKANY23AS6QT73GGLPOWDSOA, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:34.155572455 (617h54m34.0266555s) [noise=true , address=45.91.116.10:4082 , piece_id=IXNZH7SCVEUL4IIX5GXMHACPMYWOUC4WCYXAN4F6KVLRHBZ6EFEA, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:34.199646349 (617h54m34.070729384s) [noise=true , address=45.91.116.9:4264 , piece_id=AHTIUGWB42LTI37NI6TEWL2ZQ7A5354IA2V25WW7DIY3FY3I5XIQ, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:34.233994352 (617h54m34.105077407s) [noise=true , address=45.91.116.8:4047 , piece_id=YJFG3OZJYACXVQZUDJCQB5RES3HCHOHI7DUTO6WI75KJ3QDOC7VA, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:34.266343800 (617h54m34.137426855s) [noise=true , address=45.91.116.4:4940 , piece_id=DGDAW2WEIMBZJVJJKB2Z63EXSUZ67BYKDBKJ3QETAXM6VFBAQD7Q, seg_pos={0 0}, upload=541490]: Done uploading piece. err: <nil> Dec 31 09:23:34.266368006 (617h54m34.137451051s) [seg_pos={0 0}, upload=541490]: Segment reached optimal threshold of 13 pieces. Dec 31 09:23:34.266578988 (617h54m34.137662033s) [noise=true , address=45.91.116.7:4012 , piece_id=G7Z63HRUBGSN6CK4ZNWJ6QGL2XAS26QHHYAWT6WYG7NE3Q3M57PA, seg_pos={0 0}, upload=541490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:36890->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:36890->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36890->45.91.116.7:4012: use of closed network connection Dec 31 09:23:34.266587214 (617h54m34.137670249s) [noise=true , address=45.91.116.13:4156 , piece_id=2AX67JV6G5QBHSNXRXV2CVWSXRGRBI6SSQ52CWYI7HIHT2DG6R4Q, seg_pos={0 0}, upload=541490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 09:23:34.266595800 (617h54m34.137678835s) [noise=true , address=45.91.116.13:4156 , piece_id=2AX67JV6G5QBHSNXRXV2CVWSXRGRBI6SSQ52CWYI7HIHT2DG6R4Q, seg_pos={0 0}, upload=541490]: Long tail cancellation due to optimal threshold Dec 31 09:23:34.266604546 (617h54m34.137687591s) [noise=true , address=45.91.116.7:4012 , piece_id=G7Z63HRUBGSN6CK4ZNWJ6QGL2XAS26QHHYAWT6WYG7NE3Q3M57PA, seg_pos={0 0}, upload=541490]: Long tail cancellation due to optimal threshold Dec 31 09:23:34.266773899 (617h54m34.137856934s) [noise=true , address=145.79.227.44:4193 , piece_id=NF7UFT2Y4V57XMXN42KC2JIZBZT47MYHU7RDB7IQDG4CB2MKN7OQ, seg_pos={0 0}, upload=541490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 31 09:23:34.266785211 (617h54m34.137868256s) [noise=true , address=145.79.227.44:4193 , piece_id=NF7UFT2Y4V57XMXN42KC2JIZBZT47MYHU7RDB7IQDG4CB2MKN7OQ, seg_pos={0 0}, upload=541490]: Long tail cancellation due to optimal threshold Dec 31 09:23:34.266819206 (617h54m34.137902241s) [seg_pos={0 0}, upload=541490]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:23:34.266898356 (617h54m34.137981391s) [upload=541490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:23:34.266905570 (617h54m34.137988605s) [upload=541490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:23:34.557556471 (617h54m34.428639516s) [upload=541490]: Done uploading segments... Dec 31 09:23:34.557576950 (617h54m34.428659985s) [upload=541490]: Upload finished. err: <nil> Duration: 30.002717183s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...........X....X....XXX.XX.............XX...........X......] duration: 3.573346823s trace: https://jaeger.dev.storj.io/trace/71ca0a73df078f5e FAIL (2025-12-31 09:23:51 UTC) orbiter warning: context deadline exceeded Duration: 3.573368203s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X...........X....X....XXX.XX.............XX...........X.....] duration: 3.521163466s trace: https://jaeger.dev.storj.io/trace/71ca0a73df078f5e FAIL (2025-12-31 09:23:07 UTC) orbiter warning: context deadline exceeded Duration: 3.521193774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X...XXX..X.X.....X........XX....X..XX............X..X.X..XX] duration: 30.003117042s trace: https://jaeger.dev.storj.io/trace/19c594a240963021 FAIL (2025-12-31 09:22:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TCOVMKGEQDHA33H7HSDJKY5LKI32WGG32C72ILKXDEUGHKP2W2AQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: KLMSWLA5DBPYVSDGNXBPFYHTW7RSXCLNAOGG5IWJWBMGP4FUHSOA) context deadline exceeded ======LOGS====== 1 09:22:27.981737908 (617h53m27.852820943s) [upload=541475]: Starting upload Dec 31 09:22:27.981832868 (617h53m27.852915903s) [upload=541475]: Done starting upload Dec 31 09:22:27.981868396 (617h53m27.852951431s) [upload=541475]: Uploading segments... Dec 31 09:22:27.982032909 (617h53m27.853115944s) [upload=541475]: Got next segment. Inline: false Dec 31 09:22:27.982105107 (617h53m27.853188142s) [upload=541475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:22:27.982110046 (617h53m27.853193091s) [upload=541475]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:22:28.231878110 (617h53m28.102961156s) [seg_pos={0 0}, upload=541475]: Begin upload segment... Dec 31 09:22:28.231934398 (617h53m28.103017443s) [seg_pos={0 0}, upload=541475]: Done begin upload segment. Dec 31 09:22:28.231970787 (617h53m28.103053822s) [noise=true , address=77.90.31.10:4520 , piece_id=IHYYLMSCRQX52AUNQ6XCEPW2LF7PZME7NJNP4VMFSJEEOZ2JJSFA, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.232007637 (617h53m28.103090682s) [noise=true , address=77.90.31.13:4629 , piece_id=R7UI7M6SZE6TBPXDLCDT3FGDXYLMAXIS3R5O2JYTEY32UGBOIPTQ, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.232049697 (617h53m28.103132732s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMSWLA5DBPYVSDGNXBPFYHTW7RSXCLNAOGG5IWJWBMGP4FUHSOA, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.232586168 (617h53m28.103669213s) [noise=true , address=145.79.227.33:4871 , piece_id=YU6572GFSDG2YTYXZL354G4PUIMM6R4JTQTVET6MMUBYTFHADGDA, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.232658246 (617h53m28.103741281s) [noise=true , address=145.79.227.47:4265 , piece_id=DLGLZ4YGJTFXDIRTWURZBFUG4ROPAMFZIDGSRUF7BJVEJFO6SWLQ, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.233000778 (617h53m28.104083813s) [noise=true , address=145.79.227.2:4115 , piece_id=V6KHM2XT3ISJGEZNXLO65L2MPZSA4GOBR5RJBOAUV3X6XBSQZB6A, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.233045693 (617h53m28.104128728s) [noise=true , address=145.79.227.17:4970 , piece_id=IJOCFJAFBBQKNEKZFTFR4CGGG2NOTOO5WLATSEH33RL3PN7KEC6A, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.233396753 (617h53m28.104479798s) [noise=true , address=145.79.227.13:4803 , piece_id=VUVPXEJIERL2T2SD7NXVH5SKZ525BOBAHDOXL37KVEPUPKMVKM7Q, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.233464331 (617h53m28.104547376s) [noise=true , address=45.91.116.13:4156 , piece_id=Z27V2H7ZAOZA6SELRTJFJYHG3WBY5LEP2CI5OH6TTH4S2NU4WNTQ, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.233867208 (617h53m28.104950253s) [noise=true , address=145.79.227.51:4409 , piece_id=LAOEAWZY6YPS553S7AYS6DPCP2SBV2FQSJFRRMS3LJAJIRV37CHA, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.234145067 (617h53m28.105228102s) [noise=true , address=145.79.227.24:4583 , piece_id=EOPQP7IOGNTJ366VB7WBRJRNCVWWN4FHLUJU4HEAACW6IXJDSPMQ, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.234386006 (617h53m28.105469051s) [noise=true , address=145.79.227.37:4566 , piece_id=3GSFTAD2AA533RXWIFCVBCQHZ7DLTYE5HO6GMU6VA23IPGZRSTZA, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.234663224 (617h53m28.105746269s) [noise=true , address=45.91.116.12:4227 , piece_id=5Z57MLWWB5SY7J4WE3VJD6RRNQOJCZVOBZMO4SP2CEGKQH7T5SZA, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.234844910 (617h53m28.105927945s) [noise=true , address=145.79.227.38:4979 , piece_id=DAURRXAUCKSI6Y3YAQYO5TOLURWM4SSIHYO26NBQTHAC7CMQXCVA, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.235440534 (617h53m28.106523579s) [noise=true , address=145.79.227.43:4158 , piece_id=3FRPRD7DL7RHBJSXNWBA6KU46UYREZAK6RRTSQOTCLF3LUNZQG4A, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:28.235826218 (617h53m28.106909263s) [noise=true , address=77.90.31.11:4554 , piece_id=TCOVMKGEQDHA33H7HSDJKY5LKI32WGG32C72ILKXDEUGHKP2W2AQ, seg_pos={0 0}, upload=541475]: Uploading piece... Dec 31 09:22:29.937041085 (617h53m29.808124131s) [upload=541475]: Next returned nil segment Dec 31 09:22:29.937055162 (617h53m29.808138197s) [upload=541475]: Waiting for error group managing segments... Dec 31 09:22:30.218158453 (617h53m30.089241518s) [noise=true , address=145.79.227.2:4115 , piece_id=V6KHM2XT3ISJGEZNXLO65L2MPZSA4GOBR5RJBOAUV3X6XBSQZB6A, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.237176075 (617h53m30.10825912s) [noise=true , address=145.79.227.13:4803 , piece_id=VUVPXEJIERL2T2SD7NXVH5SKZ525BOBAHDOXL37KVEPUPKMVKM7Q, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.238678827 (617h53m30.109761862s) [noise=true , address=145.79.227.24:4583 , piece_id=EOPQP7IOGNTJ366VB7WBRJRNCVWWN4FHLUJU4HEAACW6IXJDSPMQ, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.240774828 (617h53m30.111857863s) [noise=true , address=145.79.227.37:4566 , piece_id=3GSFTAD2AA533RXWIFCVBCQHZ7DLTYE5HO6GMU6VA23IPGZRSTZA, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.242755970 (617h53m30.113839015s) [noise=true , address=145.79.227.17:4970 , piece_id=IJOCFJAFBBQKNEKZFTFR4CGGG2NOTOO5WLATSEH33RL3PN7KEC6A, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.250348597 (617h53m30.121431642s) [noise=true , address=145.79.227.38:4979 , piece_id=DAURRXAUCKSI6Y3YAQYO5TOLURWM4SSIHYO26NBQTHAC7CMQXCVA, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.302471196 (617h53m30.173554231s) [noise=true , address=145.79.227.51:4409 , piece_id=LAOEAWZY6YPS553S7AYS6DPCP2SBV2FQSJFRRMS3LJAJIRV37CHA, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.317901498 (617h53m30.188984543s) [noise=true , address=145.79.227.47:4265 , piece_id=DLGLZ4YGJTFXDIRTWURZBFUG4ROPAMFZIDGSRUF7BJVEJFO6SWLQ, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.406928978 (617h53m30.278012023s) [noise=true , address=77.90.31.11:4554 , piece_id=TCOVMKGEQDHA33H7HSDJKY5LKI32WGG32C72ILKXDEUGHKP2W2AQ, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.420481103 (617h53m30.291564148s) [noise=true , address=77.90.31.13:4629 , piece_id=R7UI7M6SZE6TBPXDLCDT3FGDXYLMAXIS3R5O2JYTEY32UGBOIPTQ, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.567393205 (617h53m30.43847625s) [noise=true , address=45.91.116.12:4227 , piece_id=5Z57MLWWB5SY7J4WE3VJD6RRNQOJCZVOBZMO4SP2CEGKQH7T5SZA, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.570006821 (617h53m30.441089856s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMSWLA5DBPYVSDGNXBPFYHTW7RSXCLNAOGG5IWJWBMGP4FUHSOA, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.625199696 (617h53m30.496282731s) [noise=true , address=77.90.31.10:4520 , piece_id=IHYYLMSCRQX52AUNQ6XCEPW2LF7PZME7NJNP4VMFSJEEOZ2JJSFA, seg_pos={0 0}, upload=541475]: Done uploading piece. err: <nil> Dec 31 09:22:30.625225074 (617h53m30.496308119s) [seg_pos={0 0}, upload=541475]: Segment reached optimal threshold of 13 pieces. Dec 31 09:22:30.625462056 (617h53m30.496545101s) [noise=true , address=145.79.227.43:4158 , piece_id=3FRPRD7DL7RHBJSXNWBA6KU46UYREZAK6RRTSQOTCLF3LUNZQG4A, seg_pos={0 0}, upload=541475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 31 09:22:30.625471253 (617h53m30.496554308s) [noise=true , address=145.79.227.43:4158 , piece_id=3FRPRD7DL7RHBJSXNWBA6KU46UYREZAK6RRTSQOTCLF3LUNZQG4A, seg_pos={0 0}, upload=541475]: Long tail cancellation due to optimal threshold Dec 31 09:22:30.625525737 (617h53m30.496608792s) [noise=true , address=45.91.116.13:4156 , piece_id=Z27V2H7ZAOZA6SELRTJFJYHG3WBY5LEP2CI5OH6TTH4S2NU4WNTQ, seg_pos={0 0}, upload=541475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:51586->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:51586->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51586->45.91.116.13:4156: use of closed network connection Dec 31 09:22:30.625536878 (617h53m30.496619933s) [noise=true , address=45.91.116.13:4156 , piece_id=Z27V2H7ZAOZA6SELRTJFJYHG3WBY5LEP2CI5OH6TTH4S2NU4WNTQ, seg_pos={0 0}, upload=541475]: Long tail cancellation due to optimal threshold Dec 31 09:22:30.625730307 (617h53m30.496813352s) [noise=true , address=145.79.227.33:4871 , piece_id=YU6572GFSDG2YTYXZL354G4PUIMM6R4JTQTVET6MMUBYTFHADGDA, seg_pos={0 0}, upload=541475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 31 09:22:30.625739935 (617h53m30.49682297s) [noise=true , address=145.79.227.33:4871 , piece_id=YU6572GFSDG2YTYXZL354G4PUIMM6R4JTQTVET6MMUBYTFHADGDA, seg_pos={0 0}, upload=541475]: Long tail cancellation due to optimal threshold Dec 31 09:22:30.625768860 (617h53m30.496851905s) [seg_pos={0 0}, upload=541475]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:22:30.625889559 (617h53m30.496972605s) [upload=541475]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:22:30.625896322 (617h53m30.496979367s) [upload=541475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:22:30.925100059 (617h53m30.796183094s) [upload=541475]: Done uploading segments... Dec 31 09:22:30.925119155 (617h53m30.7962022s) [upload=541475]: Upload finished. err: <nil> Duration: 30.003132663s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X...........X....X....XXX.XX.............XX...........X....] duration: 3.62180909s trace: https://jaeger.dev.storj.io/trace/71ca0a73df078f5e FAIL (2025-12-31 09:22:39 UTC) orbiter warning: context deadline exceeded Duration: 3.621830841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X...XXX..X.X.....X........XX....X..XX............X..X.X..X] duration: 30.002497511s trace: https://jaeger.dev.storj.io/trace/5341ded932c67137 FAIL (2025-12-31 09:22:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 09:21:48.419857250 (617h52m48.290940295s) [upload=541465]: Starting upload Dec 31 09:21:48.419941480 (617h52m48.291024515s) [upload=541465]: Done starting upload Dec 31 09:21:48.419992678 (617h52m48.291075713s) [upload=541465]: Uploading segments... Dec 31 09:21:48.420236082 (617h52m48.291319117s) [upload=541465]: Got next segment. Inline: false Dec 31 09:21:48.420284814 (617h52m48.291367849s) [upload=541465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:21:48.420291086 (617h52m48.291374131s) [upload=541465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:21:48.666523688 (617h52m48.537606733s) [seg_pos={0 0}, upload=541465]: Begin upload segment... Dec 31 09:21:48.666631553 (617h52m48.537714598s) [seg_pos={0 0}, upload=541465]: Done begin upload segment. Dec 31 09:21:48.666674765 (617h52m48.53775781s) [noise=true , address=45.91.116.6:4974 , piece_id=MHAJGCLIU6W3EDYC35AEU5JBEZ7TXD2YLT5MVB67ZIR6TP7EDQOQ, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.666720733 (617h52m48.537803768s) [noise=true , address=145.79.227.8:4330 , piece_id=LOPURTSBBBJDDV2UDOA7G4QQGVVPRP4ZPL7E5NKDWAAQRQV6L2SQ, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.667310196 (617h52m48.538393271s) [noise=true , address=145.79.227.29:4943 , piece_id=2P5GG6JR2CFTSHYZMRJERKG6HWJ32YKL3BWOCUX5IE6E6QCOQGKQ, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.667792114 (617h52m48.538875159s) [noise=true , address=77.90.31.17:4770 , piece_id=BN46JTDCBKXALLGFNB3J2MKGXFGIKC6JVHEPKWOK44X5HP7F3S4A, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.668259603 (617h52m48.539342638s) [noise=true , address=145.79.227.35:4482 , piece_id=EQJYLES34T2T63XWJBQOESEJT6M67GTU4KQH3ECTCIHIWDNHIPGQ, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.668671558 (617h52m48.539754613s) [noise=true , address=145.79.227.39:4014 , piece_id=N2ZPTEMQVX7BI55HVQTFLNQJLG3NI2RNMXHWYE6KMPKDVHWDMR2A, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.669425684 (617h52m48.540508729s) [noise=true , address=145.79.227.14:4839 , piece_id=IUO2VNKIAI4AFO7MJV4OSGXZVRTLAVBOVGVVTDM6AEXTQZRPPWHA, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.669637307 (617h52m48.540720342s) [noise=true , address=145.79.227.40:4050 , piece_id=4W4UQXQQXP2PCUS7HUDDLR2WMZSJ7SABPG3NQGYTBCDAQGAI4XRA, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.669868387 (617h52m48.540951422s) [noise=true , address=77.90.31.18:4808 , piece_id=WHWWGJ3YLKOLJVZ6HRHJ73WY7TNGXID6FYDNYU6JBZTDG6MVUZUQ, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.670882267 (617h52m48.541965312s) [noise=true , address=145.79.227.46:4601 , piece_id=FDHBYSI67VZ6UFQCFSNGD2NGBMRDZONEOJCNUUL3S445PXBJ236Q, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.671214540 (617h52m48.542297575s) [noise=true , address=77.90.31.10:4517 , piece_id=BLPLFMHJ6DTTBAKDX5FEHQPMJHLQUWTYRXP6LFX42GXN6IEDQYGQ, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.671505123 (617h52m48.542588158s) [noise=true , address=145.79.227.30:4761 , piece_id=IR4YD6RSAZ7CPRG5UL7ZYTMROQFNSVNVPPGT3DFO5XPF2434TP3Q, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.671627176 (617h52m48.542710221s) [noise=true , address=45.91.116.8:4046 , piece_id=VESIMMMYE2BLGNYEGLPBKYSTPKHNQBLDVATSRQ65JLA7RXPYN2GA, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.672198864 (617h52m48.543281909s) [noise=true , address=45.91.116.11:4118 , piece_id=DHA4D4SKTIZJHZZD6G4P77T37YPIEIFVXV2NU4PGGUTNGEX3ZLLA, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.672279619 (617h52m48.543362654s) [noise=true , address=145.79.227.19:4078 , piece_id=N53GTI5NWQKGSDWQWHR2SARONZUWAMOY3BXGLUCHDEAARXJ3FKMQ, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:48.672707924 (617h52m48.543790969s) [noise=true , address=145.79.227.10:4401 , piece_id=GW55PTC4OUI65UYMODMQW353EJDYWGBWEKM52ZPNE3U7QVLNVHOA, seg_pos={0 0}, upload=541465]: Uploading piece... Dec 31 09:21:50.366265500 (617h52m50.237348535s) [upload=541465]: Next returned nil segment Dec 31 09:21:50.366279516 (617h52m50.237362551s) [upload=541465]: Waiting for error group managing segments... Dec 31 09:21:50.657518116 (617h52m50.528601151s) [noise=true , address=145.79.227.19:4078 , piece_id=N53GTI5NWQKGSDWQWHR2SARONZUWAMOY3BXGLUCHDEAARXJ3FKMQ, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:50.657597156 (617h52m50.528680201s) [noise=true , address=145.79.227.10:4401 , piece_id=GW55PTC4OUI65UYMODMQW353EJDYWGBWEKM52ZPNE3U7QVLNVHOA, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:50.667196785 (617h52m50.53827982s) [noise=true , address=145.79.227.29:4943 , piece_id=2P5GG6JR2CFTSHYZMRJERKG6HWJ32YKL3BWOCUX5IE6E6QCOQGKQ, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:50.678815850 (617h52m50.549898885s) [noise=true , address=145.79.227.46:4601 , piece_id=FDHBYSI67VZ6UFQCFSNGD2NGBMRDZONEOJCNUUL3S445PXBJ236Q, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:50.679528336 (617h52m50.550611371s) [noise=true , address=145.79.227.8:4330 , piece_id=LOPURTSBBBJDDV2UDOA7G4QQGVVPRP4ZPL7E5NKDWAAQRQV6L2SQ, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:50.685772195 (617h52m50.55685523s) [noise=true , address=145.79.227.39:4014 , piece_id=N2ZPTEMQVX7BI55HVQTFLNQJLG3NI2RNMXHWYE6KMPKDVHWDMR2A, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:50.744216582 (617h52m50.615299628s) [noise=true , address=145.79.227.14:4839 , piece_id=IUO2VNKIAI4AFO7MJV4OSGXZVRTLAVBOVGVVTDM6AEXTQZRPPWHA, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:50.957568124 (617h52m50.828651159s) [noise=true , address=45.91.116.8:4046 , piece_id=VESIMMMYE2BLGNYEGLPBKYSTPKHNQBLDVATSRQ65JLA7RXPYN2GA, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:50.967892863 (617h52m50.838975908s) [noise=true , address=45.91.116.11:4118 , piece_id=DHA4D4SKTIZJHZZD6G4P77T37YPIEIFVXV2NU4PGGUTNGEX3ZLLA, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:50.981863395 (617h52m50.85294644s) [noise=true , address=77.90.31.18:4808 , piece_id=WHWWGJ3YLKOLJVZ6HRHJ73WY7TNGXID6FYDNYU6JBZTDG6MVUZUQ, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:51.169983113 (617h52m51.041066148s) [noise=true , address=45.91.116.6:4974 , piece_id=MHAJGCLIU6W3EDYC35AEU5JBEZ7TXD2YLT5MVB67ZIR6TP7EDQOQ, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:51.759382273 (617h52m51.630465308s) [noise=true , address=77.90.31.17:4770 , piece_id=BN46JTDCBKXALLGFNB3J2MKGXFGIKC6JVHEPKWOK44X5HP7F3S4A, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:52.063677515 (617h52m51.93476055s) [noise=true , address=145.79.227.30:4761 , piece_id=IR4YD6RSAZ7CPRG5UL7ZYTMROQFNSVNVPPGT3DFO5XPF2434TP3Q, seg_pos={0 0}, upload=541465]: Done uploading piece. err: <nil> Dec 31 09:21:52.063706811 (617h52m51.934789856s) [seg_pos={0 0}, upload=541465]: Segment reached optimal threshold of 13 pieces. Dec 31 09:21:52.063971826 (617h52m51.935054871s) [noise=true , address=145.79.227.35:4482 , piece_id=EQJYLES34T2T63XWJBQOESEJT6M67GTU4KQH3ECTCIHIWDNHIPGQ, seg_pos={0 0}, upload=541465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 31 09:21:52.063981204 (617h52m51.935064239s) [noise=true , address=145.79.227.35:4482 , piece_id=EQJYLES34T2T63XWJBQOESEJT6M67GTU4KQH3ECTCIHIWDNHIPGQ, seg_pos={0 0}, upload=541465]: Long tail cancellation due to optimal threshold Dec 31 09:21:52.064078249 (617h52m51.935161324s) [noise=true , address=145.79.227.40:4050 , piece_id=4W4UQXQQXP2PCUS7HUDDLR2WMZSJ7SABPG3NQGYTBCDAQGAI4XRA, seg_pos={0 0}, upload=541465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 31 09:21:52.064088528 (617h52m51.935171573s) [noise=true , address=145.79.227.40:4050 , piece_id=4W4UQXQQXP2PCUS7HUDDLR2WMZSJ7SABPG3NQGYTBCDAQGAI4XRA, seg_pos={0 0}, upload=541465]: Long tail cancellation due to optimal threshold Dec 31 09:21:52.064566278 (617h52m51.935649323s) [noise=true , address=77.90.31.10:4517 , piece_id=BLPLFMHJ6DTTBAKDX5FEHQPMJHLQUWTYRXP6LFX42GXN6IEDQYGQ, seg_pos={0 0}, upload=541465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:43970->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:43970->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: context canceled Dec 31 09:21:52.064575796 (617h52m51.935658831s) [noise=true , address=77.90.31.10:4517 , piece_id=BLPLFMHJ6DTTBAKDX5FEHQPMJHLQUWTYRXP6LFX42GXN6IEDQYGQ, seg_pos={0 0}, upload=541465]: Long tail cancellation due to optimal threshold Dec 31 09:21:52.064598839 (617h52m51.935681884s) [seg_pos={0 0}, upload=541465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:21:52.064682819 (617h52m51.935765855s) [upload=541465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:21:52.064688951 (617h52m51.935771996s) [upload=541465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:21:52.365578287 (617h52m52.236661332s) [upload=541465]: Done uploading segments... Dec 31 09:21:52.365603886 (617h52m52.236686921s) [upload=541465]: Upload finished. err: <nil> Duration: 30.002519211s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..X...........X....X....XXX.XX.............XX...........X...] duration: 3.686131247s trace: https://jaeger.dev.storj.io/trace/71ca0a73df078f5e FAIL (2025-12-31 09:22:12 UTC) orbiter warning: context deadline exceeded Duration: 3.68615875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X...XXX..X.X.....X........XX....X..XX............X..X.X..] duration: 5.974238816s trace: https://jaeger.dev.storj.io/trace/35723ac1418e56e3 FAIL (2025-12-31 09:21:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: CSI7JWOWWN3YJFZFB2VZDNKJQ35UEBZZCWKWGPAQ7TLX6DOF553A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: THSOQ5HAWLBJB76LKBAHVNWE37VBBL56T6IKV4AVXP7YJDDEZQ7A) context deadline exceeded ======LOGS====== 09:19:23.100352970 (617h50m22.971436005s) [upload=541426]: Starting upload Dec 31 09:19:23.100549083 (617h50m22.971632138s) [upload=541426]: Done starting upload Dec 31 09:19:23.100764795 (617h50m22.97184784s) [upload=541426]: Uploading segments... Dec 31 09:19:23.100814749 (617h50m22.971897804s) [upload=541426]: Got next segment. Inline: false Dec 31 09:19:23.100848363 (617h50m22.971931398s) [upload=541426]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:19:23.100854515 (617h50m22.97193755s) [upload=541426]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:19:23.356577429 (617h50m23.227660474s) [seg_pos={0 0}, upload=541426]: Begin upload segment... Dec 31 09:19:23.356623697 (617h50m23.227706732s) [seg_pos={0 0}, upload=541426]: Done begin upload segment. Dec 31 09:19:23.356658383 (617h50m23.227741448s) [noise=true , address=145.79.227.37:4567 , piece_id=2Q5RIIC2T5SC5RFQJZ7PQBBTKPTHX6JESD4M3Z5XAH3I2W5O3KCQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.356712697 (617h50m23.227795732s) [noise=true , address=145.79.227.24:4581 , piece_id=2CZPEEXYGMQVN3QWB6KK7JBOKMH7N2ESZVKVYQ6U67MZ76EX7STA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.357257263 (617h50m23.228340298s) [noise=true , address=145.79.227.33:4869 , piece_id=VQDGVU4QURLFJ3A27BYYMCW5U4RWGR6BDRBRDC522SOWOQWX77YA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.357503682 (617h50m23.228586717s) [noise=true , address=45.91.116.13:4155 , piece_id=XMKC5TVOFBL25R46JIYONHC2QKUQF3BS2DKBH3OBBTOG7AF7DPHA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.357744461 (617h50m23.228827506s) [noise=true , address=77.90.31.12:4589 , piece_id=THSOQ5HAWLBJB76LKBAHVNWE37VBBL56T6IKV4AVXP7YJDDEZQ7A, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358272306 (617h50m23.229355351s) [noise=true , address=145.79.227.13:4803 , piece_id=R7MWAJB4ZHOQEYPCXBORHAMTWHSQNGWW2RZHWYFMDC443WEKD4KQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358354483 (617h50m23.229437518s) [noise=true , address=145.79.227.3:4149 , piece_id=5CHKG6WCTHO5T3VA3QUUJZLQNAXR23HE4DYECHBCJLEOMOYJKETQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358678530 (617h50m23.229761565s) [noise=true , address=145.79.227.16:4912 , piece_id=PKAFECATRWSGGFVSUURC5KNDUKREMUYHS7C6SBGZWHYQGUN3MNIA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358735008 (617h50m23.229836638s) [noise=true , address=145.79.227.51:4411 , piece_id=Q4YV63POGYHLJUJLACA3XACBEG6WH23MEDCPKQNTW4PS47JU573A, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359049687 (617h50m23.230132722s) [noise=true , address=45.91.116.5:4300 , piece_id=S6BUHJ37T5OII6OGM6N4QNZT6QEELEU6THVW4W2CHTUJPF7J7XTA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359351701 (617h50m23.230434746s) [noise=true , address=45.91.116.8:4046 , piece_id=CSI7JWOWWN3YJFZFB2VZDNKJQ35UEBZZCWKWGPAQ7TLX6DOF553A, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359386638 (617h50m23.230469683s) [noise=true , address=145.79.227.32:4836 , piece_id=ZCL6NHQGOI7OPO2E6TA42W5KG7ELKIA7VRPAENIMVOLCOA4PKQAQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359726886 (617h50m23.230809921s) [noise=true , address=145.79.227.48:4302 , piece_id=DZOEOAJB7UAWQQRCE3MX6AHBDRYBHTTL2RCXX2ZDNIOE2ENSBIOQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359911477 (617h50m23.230994512s) [noise=true , address=145.79.227.12:4767 , piece_id=PYGGNJPEUM7G2W5YAGSRTWL5HN7366YKQDQNOCKRUBEV7EOPN6KQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.360139261 (617h50m23.231222296s) [noise=true , address=45.91.116.7:4009 , piece_id=OCFHX66HKETNCWHO7S7HDW4GY7REWLIK7XQIRCKEKWJLKRD7F6VQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.360528763 (617h50m23.231611798s) [noise=true , address=145.79.227.49:4338 , piece_id=TTGT66RKPSNOUFJMOLON3QPE2IEFF363I6LOBONIB4CIQC723CQQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:25.053767870 (617h50m24.924850905s) [upload=541426]: Next returned nil segment Dec 31 09:19:25.053784953 (617h50m24.924867988s) [upload=541426]: Waiting for error group managing segments... Dec 31 09:19:25.335361845 (617h50m25.206444881s) [noise=true , address=145.79.227.16:4912 , piece_id=PKAFECATRWSGGFVSUURC5KNDUKREMUYHS7C6SBGZWHYQGUN3MNIA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.341552834 (617h50m25.212635869s) [noise=true , address=145.79.227.51:4411 , piece_id=Q4YV63POGYHLJUJLACA3XACBEG6WH23MEDCPKQNTW4PS47JU573A, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.364269360 (617h50m25.235352405s) [noise=true , address=145.79.227.13:4803 , piece_id=R7MWAJB4ZHOQEYPCXBORHAMTWHSQNGWW2RZHWYFMDC443WEKD4KQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.367157059 (617h50m25.238240094s) [noise=true , address=145.79.227.32:4836 , piece_id=ZCL6NHQGOI7OPO2E6TA42W5KG7ELKIA7VRPAENIMVOLCOA4PKQAQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.371166553 (617h50m25.242249588s) [noise=true , address=145.79.227.12:4767 , piece_id=PYGGNJPEUM7G2W5YAGSRTWL5HN7366YKQDQNOCKRUBEV7EOPN6KQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.415857911 (617h50m25.286940957s) [noise=true , address=145.79.227.48:4302 , piece_id=DZOEOAJB7UAWQQRCE3MX6AHBDRYBHTTL2RCXX2ZDNIOE2ENSBIOQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.421521797 (617h50m25.292604832s) [noise=true , address=145.79.227.3:4149 , piece_id=5CHKG6WCTHO5T3VA3QUUJZLQNAXR23HE4DYECHBCJLEOMOYJKETQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.433376961 (617h50m25.304459996s) [noise=true , address=145.79.227.24:4581 , piece_id=2CZPEEXYGMQVN3QWB6KK7JBOKMH7N2ESZVKVYQ6U67MZ76EX7STA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.517278101 (617h50m25.388361146s) [noise=true , address=145.79.227.37:4567 , piece_id=2Q5RIIC2T5SC5RFQJZ7PQBBTKPTHX6JESD4M3Z5XAH3I2W5O3KCQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.520711688 (617h50m25.391794723s) [noise=true , address=77.90.31.12:4589 , piece_id=THSOQ5HAWLBJB76LKBAHVNWE37VBBL56T6IKV4AVXP7YJDDEZQ7A, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.649113283 (617h50m25.520196318s) [noise=true , address=145.79.227.49:4338 , piece_id=TTGT66RKPSNOUFJMOLON3QPE2IEFF363I6LOBONIB4CIQC723CQQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.651400158 (617h50m25.522483203s) [noise=true , address=145.79.227.33:4869 , piece_id=VQDGVU4QURLFJ3A27BYYMCW5U4RWGR6BDRBRDC522SOWOQWX77YA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.666779182 (617h50m25.537862217s) [noise=true , address=45.91.116.8:4046 , piece_id=CSI7JWOWWN3YJFZFB2VZDNKJQ35UEBZZCWKWGPAQ7TLX6DOF553A, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.666798008 (617h50m25.537881043s) [seg_pos={0 0}, upload=541426]: Segment reached optimal threshold of 13 pieces. Dec 31 09:19:25.667033856 (617h50m25.538116901s) [noise=true , address=45.91.116.5:4300 , piece_id=S6BUHJ37T5OII6OGM6N4QNZT6QEELEU6THVW4W2CHTUJPF7J7XTA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 31 09:19:25.667041301 (617h50m25.538124336s) [noise=true , address=45.91.116.5:4300 , piece_id=S6BUHJ37T5OII6OGM6N4QNZT6QEELEU6THVW4W2CHTUJPF7J7XTA, seg_pos={0 0}, upload=541426]: Long tail cancellation due to optimal threshold Dec 31 09:19:25.667242293 (617h50m25.538325338s) [noise=true , address=45.91.116.13:4155 , piece_id=XMKC5TVOFBL25R46JIYONHC2QKUQF3BS2DKBH3OBBTOG7AF7DPHA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 09:19:25.667249007 (617h50m25.538332042s) [noise=true , address=45.91.116.13:4155 , piece_id=XMKC5TVOFBL25R46JIYONHC2QKUQF3BS2DKBH3OBBTOG7AF7DPHA, seg_pos={0 0}, upload=541426]: Long tail cancellation due to optimal threshold Dec 31 09:19:25.667468995 (617h50m25.538552031s) [noise=true , address=45.91.116.7:4009 , piece_id=OCFHX66HKETNCWHO7S7HDW4GY7REWLIK7XQIRCKEKWJLKRD7F6VQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:34142->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:34142->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34142->45.91.116.7:4009: use of closed network connection Dec 31 09:19:25.667489163 (617h50m25.538572218s) [noise=true , address=45.91.116.7:4009 , piece_id=OCFHX66HKETNCWHO7S7HDW4GY7REWLIK7XQIRCKEKWJLKRD7F6VQ, seg_pos={0 0}, upload=541426]: Long tail cancellation due to optimal threshold Dec 31 09:19:25.667520313 (617h50m25.538603348s) [seg_pos={0 0}, upload=541426]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:19:25.667676962 (617h50m25.538760007s) [upload=541426]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:19:25.667685538 (617h50m25.538768583s) [upload=541426]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:19:25.963740829 (617h50m25.834823864s) [upload=541426]: Done uploading segments... Dec 31 09:19:25.963788059 (617h50m25.834871104s) [upload=541426]: Upload finished. err: <nil> Duration: 5.974268783s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...X...........X....X....XXX.XX.............XX...........X..] duration: 3.586367945s trace: https://jaeger.dev.storj.io/trace/71ca0a73df078f5e FAIL (2025-12-31 09:21:04 UTC) orbiter warning: context deadline exceeded Duration: 3.586383577s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 43.282099892s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:20:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 43.282126331s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [....X...........X....X....XXX.XX.............XX...........X.] duration: 3.630730328s trace: https://jaeger.dev.storj.io/trace/71ca0a73df078f5e FAIL (2025-12-31 09:20:45 UTC) orbiter warning: context deadline exceeded Duration: 3.630753653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X...XXX..X.X.....X........XX....X..XX............X..X.X.] duration: 6.617099893s trace: https://jaeger.dev.storj.io/trace/35723ac1418e56e3 FAIL (2025-12-31 09:20:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: CSI7JWOWWN3YJFZFB2VZDNKJQ35UEBZZCWKWGPAQ7TLX6DOF553A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: THSOQ5HAWLBJB76LKBAHVNWE37VBBL56T6IKV4AVXP7YJDDEZQ7A) context deadline exceeded ======LOGS====== 09:19:23.100352970 (617h50m22.971436005s) [upload=541426]: Starting upload Dec 31 09:19:23.100549083 (617h50m22.971632138s) [upload=541426]: Done starting upload Dec 31 09:19:23.100764795 (617h50m22.97184784s) [upload=541426]: Uploading segments... Dec 31 09:19:23.100814749 (617h50m22.971897804s) [upload=541426]: Got next segment. Inline: false Dec 31 09:19:23.100848363 (617h50m22.971931398s) [upload=541426]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:19:23.100854515 (617h50m22.97193755s) [upload=541426]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:19:23.356577429 (617h50m23.227660474s) [seg_pos={0 0}, upload=541426]: Begin upload segment... Dec 31 09:19:23.356623697 (617h50m23.227706732s) [seg_pos={0 0}, upload=541426]: Done begin upload segment. Dec 31 09:19:23.356658383 (617h50m23.227741448s) [noise=true , address=145.79.227.37:4567 , piece_id=2Q5RIIC2T5SC5RFQJZ7PQBBTKPTHX6JESD4M3Z5XAH3I2W5O3KCQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.356712697 (617h50m23.227795732s) [noise=true , address=145.79.227.24:4581 , piece_id=2CZPEEXYGMQVN3QWB6KK7JBOKMH7N2ESZVKVYQ6U67MZ76EX7STA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.357257263 (617h50m23.228340298s) [noise=true , address=145.79.227.33:4869 , piece_id=VQDGVU4QURLFJ3A27BYYMCW5U4RWGR6BDRBRDC522SOWOQWX77YA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.357503682 (617h50m23.228586717s) [noise=true , address=45.91.116.13:4155 , piece_id=XMKC5TVOFBL25R46JIYONHC2QKUQF3BS2DKBH3OBBTOG7AF7DPHA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.357744461 (617h50m23.228827506s) [noise=true , address=77.90.31.12:4589 , piece_id=THSOQ5HAWLBJB76LKBAHVNWE37VBBL56T6IKV4AVXP7YJDDEZQ7A, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358272306 (617h50m23.229355351s) [noise=true , address=145.79.227.13:4803 , piece_id=R7MWAJB4ZHOQEYPCXBORHAMTWHSQNGWW2RZHWYFMDC443WEKD4KQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358354483 (617h50m23.229437518s) [noise=true , address=145.79.227.3:4149 , piece_id=5CHKG6WCTHO5T3VA3QUUJZLQNAXR23HE4DYECHBCJLEOMOYJKETQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358678530 (617h50m23.229761565s) [noise=true , address=145.79.227.16:4912 , piece_id=PKAFECATRWSGGFVSUURC5KNDUKREMUYHS7C6SBGZWHYQGUN3MNIA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358735008 (617h50m23.229836638s) [noise=true , address=145.79.227.51:4411 , piece_id=Q4YV63POGYHLJUJLACA3XACBEG6WH23MEDCPKQNTW4PS47JU573A, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359049687 (617h50m23.230132722s) [noise=true , address=45.91.116.5:4300 , piece_id=S6BUHJ37T5OII6OGM6N4QNZT6QEELEU6THVW4W2CHTUJPF7J7XTA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359351701 (617h50m23.230434746s) [noise=true , address=45.91.116.8:4046 , piece_id=CSI7JWOWWN3YJFZFB2VZDNKJQ35UEBZZCWKWGPAQ7TLX6DOF553A, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359386638 (617h50m23.230469683s) [noise=true , address=145.79.227.32:4836 , piece_id=ZCL6NHQGOI7OPO2E6TA42W5KG7ELKIA7VRPAENIMVOLCOA4PKQAQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359726886 (617h50m23.230809921s) [noise=true , address=145.79.227.48:4302 , piece_id=DZOEOAJB7UAWQQRCE3MX6AHBDRYBHTTL2RCXX2ZDNIOE2ENSBIOQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359911477 (617h50m23.230994512s) [noise=true , address=145.79.227.12:4767 , piece_id=PYGGNJPEUM7G2W5YAGSRTWL5HN7366YKQDQNOCKRUBEV7EOPN6KQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.360139261 (617h50m23.231222296s) [noise=true , address=45.91.116.7:4009 , piece_id=OCFHX66HKETNCWHO7S7HDW4GY7REWLIK7XQIRCKEKWJLKRD7F6VQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.360528763 (617h50m23.231611798s) [noise=true , address=145.79.227.49:4338 , piece_id=TTGT66RKPSNOUFJMOLON3QPE2IEFF363I6LOBONIB4CIQC723CQQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:25.053767870 (617h50m24.924850905s) [upload=541426]: Next returned nil segment Dec 31 09:19:25.053784953 (617h50m24.924867988s) [upload=541426]: Waiting for error group managing segments... Dec 31 09:19:25.335361845 (617h50m25.206444881s) [noise=true , address=145.79.227.16:4912 , piece_id=PKAFECATRWSGGFVSUURC5KNDUKREMUYHS7C6SBGZWHYQGUN3MNIA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.341552834 (617h50m25.212635869s) [noise=true , address=145.79.227.51:4411 , piece_id=Q4YV63POGYHLJUJLACA3XACBEG6WH23MEDCPKQNTW4PS47JU573A, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.364269360 (617h50m25.235352405s) [noise=true , address=145.79.227.13:4803 , piece_id=R7MWAJB4ZHOQEYPCXBORHAMTWHSQNGWW2RZHWYFMDC443WEKD4KQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.367157059 (617h50m25.238240094s) [noise=true , address=145.79.227.32:4836 , piece_id=ZCL6NHQGOI7OPO2E6TA42W5KG7ELKIA7VRPAENIMVOLCOA4PKQAQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.371166553 (617h50m25.242249588s) [noise=true , address=145.79.227.12:4767 , piece_id=PYGGNJPEUM7G2W5YAGSRTWL5HN7366YKQDQNOCKRUBEV7EOPN6KQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.415857911 (617h50m25.286940957s) [noise=true , address=145.79.227.48:4302 , piece_id=DZOEOAJB7UAWQQRCE3MX6AHBDRYBHTTL2RCXX2ZDNIOE2ENSBIOQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.421521797 (617h50m25.292604832s) [noise=true , address=145.79.227.3:4149 , piece_id=5CHKG6WCTHO5T3VA3QUUJZLQNAXR23HE4DYECHBCJLEOMOYJKETQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.433376961 (617h50m25.304459996s) [noise=true , address=145.79.227.24:4581 , piece_id=2CZPEEXYGMQVN3QWB6KK7JBOKMH7N2ESZVKVYQ6U67MZ76EX7STA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.517278101 (617h50m25.388361146s) [noise=true , address=145.79.227.37:4567 , piece_id=2Q5RIIC2T5SC5RFQJZ7PQBBTKPTHX6JESD4M3Z5XAH3I2W5O3KCQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.520711688 (617h50m25.391794723s) [noise=true , address=77.90.31.12:4589 , piece_id=THSOQ5HAWLBJB76LKBAHVNWE37VBBL56T6IKV4AVXP7YJDDEZQ7A, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.649113283 (617h50m25.520196318s) [noise=true , address=145.79.227.49:4338 , piece_id=TTGT66RKPSNOUFJMOLON3QPE2IEFF363I6LOBONIB4CIQC723CQQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.651400158 (617h50m25.522483203s) [noise=true , address=145.79.227.33:4869 , piece_id=VQDGVU4QURLFJ3A27BYYMCW5U4RWGR6BDRBRDC522SOWOQWX77YA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.666779182 (617h50m25.537862217s) [noise=true , address=45.91.116.8:4046 , piece_id=CSI7JWOWWN3YJFZFB2VZDNKJQ35UEBZZCWKWGPAQ7TLX6DOF553A, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.666798008 (617h50m25.537881043s) [seg_pos={0 0}, upload=541426]: Segment reached optimal threshold of 13 pieces. Dec 31 09:19:25.667033856 (617h50m25.538116901s) [noise=true , address=45.91.116.5:4300 , piece_id=S6BUHJ37T5OII6OGM6N4QNZT6QEELEU6THVW4W2CHTUJPF7J7XTA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 31 09:19:25.667041301 (617h50m25.538124336s) [noise=true , address=45.91.116.5:4300 , piece_id=S6BUHJ37T5OII6OGM6N4QNZT6QEELEU6THVW4W2CHTUJPF7J7XTA, seg_pos={0 0}, upload=541426]: Long tail cancellation due to optimal threshold Dec 31 09:19:25.667242293 (617h50m25.538325338s) [noise=true , address=45.91.116.13:4155 , piece_id=XMKC5TVOFBL25R46JIYONHC2QKUQF3BS2DKBH3OBBTOG7AF7DPHA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 09:19:25.667249007 (617h50m25.538332042s) [noise=true , address=45.91.116.13:4155 , piece_id=XMKC5TVOFBL25R46JIYONHC2QKUQF3BS2DKBH3OBBTOG7AF7DPHA, seg_pos={0 0}, upload=541426]: Long tail cancellation due to optimal threshold Dec 31 09:19:25.667468995 (617h50m25.538552031s) [noise=true , address=45.91.116.7:4009 , piece_id=OCFHX66HKETNCWHO7S7HDW4GY7REWLIK7XQIRCKEKWJLKRD7F6VQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:34142->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:34142->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34142->45.91.116.7:4009: use of closed network connection Dec 31 09:19:25.667489163 (617h50m25.538572218s) [noise=true , address=45.91.116.7:4009 , piece_id=OCFHX66HKETNCWHO7S7HDW4GY7REWLIK7XQIRCKEKWJLKRD7F6VQ, seg_pos={0 0}, upload=541426]: Long tail cancellation due to optimal threshold Dec 31 09:19:25.667520313 (617h50m25.538603348s) [seg_pos={0 0}, upload=541426]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:19:25.667676962 (617h50m25.538760007s) [upload=541426]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:19:25.667685538 (617h50m25.538768583s) [upload=541426]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:19:25.963740829 (617h50m25.834823864s) [upload=541426]: Done uploading segments... Dec 31 09:19:25.963788059 (617h50m25.834871104s) [upload=541426]: Upload finished. err: <nil> Duration: 6.617133518s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.....X...........X....X....XXX.XX.............XX...........X] duration: 5.001592834s trace: https://jaeger.dev.storj.io/trace/71ca0a73df078f5e FAIL (2025-12-31 09:20:03 UTC) orbiter warning: context deadline exceeded Duration: 5.001608223s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X....X...XXX..X.X.....X........XX....X..XX............X..X.X] duration: 30.004142593s trace: https://jaeger.dev.storj.io/trace/35723ac1418e56e3 FAIL (2025-12-31 09:19:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: CSI7JWOWWN3YJFZFB2VZDNKJQ35UEBZZCWKWGPAQ7TLX6DOF553A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: THSOQ5HAWLBJB76LKBAHVNWE37VBBL56T6IKV4AVXP7YJDDEZQ7A) context deadline exceeded ======LOGS====== 09:19:23.100352970 (617h50m22.971436005s) [upload=541426]: Starting upload Dec 31 09:19:23.100549083 (617h50m22.971632138s) [upload=541426]: Done starting upload Dec 31 09:19:23.100764795 (617h50m22.97184784s) [upload=541426]: Uploading segments... Dec 31 09:19:23.100814749 (617h50m22.971897804s) [upload=541426]: Got next segment. Inline: false Dec 31 09:19:23.100848363 (617h50m22.971931398s) [upload=541426]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:19:23.100854515 (617h50m22.97193755s) [upload=541426]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:19:23.356577429 (617h50m23.227660474s) [seg_pos={0 0}, upload=541426]: Begin upload segment... Dec 31 09:19:23.356623697 (617h50m23.227706732s) [seg_pos={0 0}, upload=541426]: Done begin upload segment. Dec 31 09:19:23.356658383 (617h50m23.227741448s) [noise=true , address=145.79.227.37:4567 , piece_id=2Q5RIIC2T5SC5RFQJZ7PQBBTKPTHX6JESD4M3Z5XAH3I2W5O3KCQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.356712697 (617h50m23.227795732s) [noise=true , address=145.79.227.24:4581 , piece_id=2CZPEEXYGMQVN3QWB6KK7JBOKMH7N2ESZVKVYQ6U67MZ76EX7STA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.357257263 (617h50m23.228340298s) [noise=true , address=145.79.227.33:4869 , piece_id=VQDGVU4QURLFJ3A27BYYMCW5U4RWGR6BDRBRDC522SOWOQWX77YA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.357503682 (617h50m23.228586717s) [noise=true , address=45.91.116.13:4155 , piece_id=XMKC5TVOFBL25R46JIYONHC2QKUQF3BS2DKBH3OBBTOG7AF7DPHA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.357744461 (617h50m23.228827506s) [noise=true , address=77.90.31.12:4589 , piece_id=THSOQ5HAWLBJB76LKBAHVNWE37VBBL56T6IKV4AVXP7YJDDEZQ7A, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358272306 (617h50m23.229355351s) [noise=true , address=145.79.227.13:4803 , piece_id=R7MWAJB4ZHOQEYPCXBORHAMTWHSQNGWW2RZHWYFMDC443WEKD4KQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358354483 (617h50m23.229437518s) [noise=true , address=145.79.227.3:4149 , piece_id=5CHKG6WCTHO5T3VA3QUUJZLQNAXR23HE4DYECHBCJLEOMOYJKETQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358678530 (617h50m23.229761565s) [noise=true , address=145.79.227.16:4912 , piece_id=PKAFECATRWSGGFVSUURC5KNDUKREMUYHS7C6SBGZWHYQGUN3MNIA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.358735008 (617h50m23.229836638s) [noise=true , address=145.79.227.51:4411 , piece_id=Q4YV63POGYHLJUJLACA3XACBEG6WH23MEDCPKQNTW4PS47JU573A, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359049687 (617h50m23.230132722s) [noise=true , address=45.91.116.5:4300 , piece_id=S6BUHJ37T5OII6OGM6N4QNZT6QEELEU6THVW4W2CHTUJPF7J7XTA, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359351701 (617h50m23.230434746s) [noise=true , address=45.91.116.8:4046 , piece_id=CSI7JWOWWN3YJFZFB2VZDNKJQ35UEBZZCWKWGPAQ7TLX6DOF553A, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359386638 (617h50m23.230469683s) [noise=true , address=145.79.227.32:4836 , piece_id=ZCL6NHQGOI7OPO2E6TA42W5KG7ELKIA7VRPAENIMVOLCOA4PKQAQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359726886 (617h50m23.230809921s) [noise=true , address=145.79.227.48:4302 , piece_id=DZOEOAJB7UAWQQRCE3MX6AHBDRYBHTTL2RCXX2ZDNIOE2ENSBIOQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.359911477 (617h50m23.230994512s) [noise=true , address=145.79.227.12:4767 , piece_id=PYGGNJPEUM7G2W5YAGSRTWL5HN7366YKQDQNOCKRUBEV7EOPN6KQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.360139261 (617h50m23.231222296s) [noise=true , address=45.91.116.7:4009 , piece_id=OCFHX66HKETNCWHO7S7HDW4GY7REWLIK7XQIRCKEKWJLKRD7F6VQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:23.360528763 (617h50m23.231611798s) [noise=true , address=145.79.227.49:4338 , piece_id=TTGT66RKPSNOUFJMOLON3QPE2IEFF363I6LOBONIB4CIQC723CQQ, seg_pos={0 0}, upload=541426]: Uploading piece... Dec 31 09:19:25.053767870 (617h50m24.924850905s) [upload=541426]: Next returned nil segment Dec 31 09:19:25.053784953 (617h50m24.924867988s) [upload=541426]: Waiting for error group managing segments... Dec 31 09:19:25.335361845 (617h50m25.206444881s) [noise=true , address=145.79.227.16:4912 , piece_id=PKAFECATRWSGGFVSUURC5KNDUKREMUYHS7C6SBGZWHYQGUN3MNIA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.341552834 (617h50m25.212635869s) [noise=true , address=145.79.227.51:4411 , piece_id=Q4YV63POGYHLJUJLACA3XACBEG6WH23MEDCPKQNTW4PS47JU573A, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.364269360 (617h50m25.235352405s) [noise=true , address=145.79.227.13:4803 , piece_id=R7MWAJB4ZHOQEYPCXBORHAMTWHSQNGWW2RZHWYFMDC443WEKD4KQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.367157059 (617h50m25.238240094s) [noise=true , address=145.79.227.32:4836 , piece_id=ZCL6NHQGOI7OPO2E6TA42W5KG7ELKIA7VRPAENIMVOLCOA4PKQAQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.371166553 (617h50m25.242249588s) [noise=true , address=145.79.227.12:4767 , piece_id=PYGGNJPEUM7G2W5YAGSRTWL5HN7366YKQDQNOCKRUBEV7EOPN6KQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.415857911 (617h50m25.286940957s) [noise=true , address=145.79.227.48:4302 , piece_id=DZOEOAJB7UAWQQRCE3MX6AHBDRYBHTTL2RCXX2ZDNIOE2ENSBIOQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.421521797 (617h50m25.292604832s) [noise=true , address=145.79.227.3:4149 , piece_id=5CHKG6WCTHO5T3VA3QUUJZLQNAXR23HE4DYECHBCJLEOMOYJKETQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.433376961 (617h50m25.304459996s) [noise=true , address=145.79.227.24:4581 , piece_id=2CZPEEXYGMQVN3QWB6KK7JBOKMH7N2ESZVKVYQ6U67MZ76EX7STA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.517278101 (617h50m25.388361146s) [noise=true , address=145.79.227.37:4567 , piece_id=2Q5RIIC2T5SC5RFQJZ7PQBBTKPTHX6JESD4M3Z5XAH3I2W5O3KCQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.520711688 (617h50m25.391794723s) [noise=true , address=77.90.31.12:4589 , piece_id=THSOQ5HAWLBJB76LKBAHVNWE37VBBL56T6IKV4AVXP7YJDDEZQ7A, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.649113283 (617h50m25.520196318s) [noise=true , address=145.79.227.49:4338 , piece_id=TTGT66RKPSNOUFJMOLON3QPE2IEFF363I6LOBONIB4CIQC723CQQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.651400158 (617h50m25.522483203s) [noise=true , address=145.79.227.33:4869 , piece_id=VQDGVU4QURLFJ3A27BYYMCW5U4RWGR6BDRBRDC522SOWOQWX77YA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.666779182 (617h50m25.537862217s) [noise=true , address=45.91.116.8:4046 , piece_id=CSI7JWOWWN3YJFZFB2VZDNKJQ35UEBZZCWKWGPAQ7TLX6DOF553A, seg_pos={0 0}, upload=541426]: Done uploading piece. err: <nil> Dec 31 09:19:25.666798008 (617h50m25.537881043s) [seg_pos={0 0}, upload=541426]: Segment reached optimal threshold of 13 pieces. Dec 31 09:19:25.667033856 (617h50m25.538116901s) [noise=true , address=45.91.116.5:4300 , piece_id=S6BUHJ37T5OII6OGM6N4QNZT6QEELEU6THVW4W2CHTUJPF7J7XTA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 31 09:19:25.667041301 (617h50m25.538124336s) [noise=true , address=45.91.116.5:4300 , piece_id=S6BUHJ37T5OII6OGM6N4QNZT6QEELEU6THVW4W2CHTUJPF7J7XTA, seg_pos={0 0}, upload=541426]: Long tail cancellation due to optimal threshold Dec 31 09:19:25.667242293 (617h50m25.538325338s) [noise=true , address=45.91.116.13:4155 , piece_id=XMKC5TVOFBL25R46JIYONHC2QKUQF3BS2DKBH3OBBTOG7AF7DPHA, seg_pos={0 0}, upload=541426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 09:19:25.667249007 (617h50m25.538332042s) [noise=true , address=45.91.116.13:4155 , piece_id=XMKC5TVOFBL25R46JIYONHC2QKUQF3BS2DKBH3OBBTOG7AF7DPHA, seg_pos={0 0}, upload=541426]: Long tail cancellation due to optimal threshold Dec 31 09:19:25.667468995 (617h50m25.538552031s) [noise=true , address=45.91.116.7:4009 , piece_id=OCFHX66HKETNCWHO7S7HDW4GY7REWLIK7XQIRCKEKWJLKRD7F6VQ, seg_pos={0 0}, upload=541426]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:34142->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:34142->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34142->45.91.116.7:4009: use of closed network connection Dec 31 09:19:25.667489163 (617h50m25.538572218s) [noise=true , address=45.91.116.7:4009 , piece_id=OCFHX66HKETNCWHO7S7HDW4GY7REWLIK7XQIRCKEKWJLKRD7F6VQ, seg_pos={0 0}, upload=541426]: Long tail cancellation due to optimal threshold Dec 31 09:19:25.667520313 (617h50m25.538603348s) [seg_pos={0 0}, upload=541426]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:19:25.667676962 (617h50m25.538760007s) [upload=541426]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:19:25.667685538 (617h50m25.538768583s) [upload=541426]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:19:25.963740829 (617h50m25.834823864s) [upload=541426]: Done uploading segments... Dec 31 09:19:25.963788059 (617h50m25.834871104s) [upload=541426]: Upload finished. err: <nil> Duration: 30.004165085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 15.357189675s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:19:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 15.357216605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X....X...XXX..X.X.....X........XX....X..XX............X..X.] duration: 6.248414786s trace: https://jaeger.dev.storj.io/trace/900c491f1cc7c8e FAIL (2025-12-31 09:19:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 09:18:20.587728627 (617h49m20.458811672s) [upload=541411]: Starting upload Dec 31 09:18:20.587804792 (617h49m20.458887827s) [upload=541411]: Done starting upload Dec 31 09:18:20.587874294 (617h49m20.458957329s) [upload=541411]: Uploading segments... Dec 31 09:18:20.588063024 (617h49m20.459146059s) [upload=541411]: Got next segment. Inline: false Dec 31 09:18:20.588122246 (617h49m20.459205291s) [upload=541411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:18:20.588128528 (617h49m20.459211573s) [upload=541411]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:18:20.837523231 (617h49m20.708606266s) [seg_pos={0 0}, upload=541411]: Begin upload segment... Dec 31 09:18:20.837588926 (617h49m20.708671971s) [seg_pos={0 0}, upload=541411]: Done begin upload segment. Dec 31 09:18:20.837636156 (617h49m20.708719191s) [noise=true , address=45.91.116.9:4263 , piece_id=WEA4K2VUOA7437THKT62Q3JHKRBWOLZYX2AOIAI2ICLKCU5I7XIA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.837643479 (617h49m20.708726515s) [noise=true , address=145.79.227.32:4834 , piece_id=OEM2CWGUEEVM546YPRHPXPK3375OHGR4H5EDDYFJBMKP4KY5UOGQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.838186404 (617h49m20.709269449s) [noise=true , address=45.91.116.10:4084 , piece_id=MV3HSJIRD3OLI7CSWDGTJWT6R446YKO5KVEPQKITEDHYLJ7F43FA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.838516232 (617h49m20.709599277s) [noise=true , address=45.91.116.7:4011 , piece_id=I27VVLYUQZPYAC7AHR6QWIZUQSSWU2JK364CLEIHKGPZBLCVRHJQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.838844076 (617h49m20.709927121s) [noise=true , address=145.79.227.27:4690 , piece_id=TYGNBB2LNEGANSUW7I3YDAKLXK7CDIIB36SUJSOH77VW2VF5IVIQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.839173824 (617h49m20.710256859s) [noise=true , address=145.79.227.46:4603 , piece_id=K7L7CU72XDU3NNUJN7EHOEK3Q6OYGEOWMC3KLT53SASDDNZEDS3A, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.839595557 (617h49m20.710678603s) [noise=true , address=45.91.116.8:4046 , piece_id=23TJYJ2N55A4VIVDJRZRHTIQIZKEV5TXEIIBUITVOTNFDOGE5QTA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.840020647 (617h49m20.711103682s) [noise=true , address=145.79.227.52:4447 , piece_id=5TINLO2QUFFV2UNMMY65BAQKQJZJ4HHX5MGSEJHA7DBV73KBDSMA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.840521220 (617h49m20.711604265s) [noise=true , address=45.91.116.5:4299 , piece_id=ZI66OQ7OLIGLDP35Z3KD6PPDX6DU7ZA6I4FE7Y3HRBVGXQYGGOWQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.840532150 (617h49m20.711615185s) [noise=true , address=145.79.227.19:4079 , piece_id=XUZOEXYVA2W5V7CKBYLV6RJKMIT3HL3QH54THTYLDZGPP4AXIM4A, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.840877878 (617h49m20.711960923s) [noise=true , address=145.79.227.16:4911 , piece_id=PQOKWZ6NSQ7J2TZCRC6HLHASQGSGFCHNWEEVLX4U3FJOZ3ZE3ULA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.840988410 (617h49m20.712071455s) [noise=true , address=145.79.227.22:4511 , piece_id=JBK7Q3VEQTMXWKCTSYS64WZ6GWJ3B6P5CDTYHX4IHB5EHJQCSECA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.841400774 (617h49m20.712483809s) [noise=true , address=145.79.227.9:4365 , piece_id=ILH4QZ27CRQEMZYJDVHQSHYXK7HS6AYEMHMTTZ3ORKGJVY5QPWRQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.841864347 (617h49m20.712947393s) [noise=true , address=145.79.227.28:4725 , piece_id=RDLMTDF7IOPHLMBLXHPUG3Y5ZHLBJ4PORIOP7ULPXXUBB7XIUTIQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.842369379 (617h49m20.713452414s) [noise=true , address=145.79.227.51:4411 , piece_id=5SB3LW7HETIL2GTOFIIFYXJHOFNV4FFQX3AGGVMGJCRPUYHSCM4A, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.842463137 (617h49m20.713546183s) [noise=true , address=77.90.31.13:4627 , piece_id=GOCQCQVDWBMB4HOMBOG75DCU6C7ZMMZUNOMG6O4UJDHFODD5HVRA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:22.539327069 (617h49m22.410410114s) [upload=541411]: Next returned nil segment Dec 31 09:18:22.539345043 (617h49m22.410428088s) [upload=541411]: Waiting for error group managing segments... Dec 31 09:18:22.827003025 (617h49m22.69808606s) [noise=true , address=145.79.227.46:4603 , piece_id=K7L7CU72XDU3NNUJN7EHOEK3Q6OYGEOWMC3KLT53SASDDNZEDS3A, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.827576277 (617h49m22.698659322s) [noise=true , address=145.79.227.9:4365 , piece_id=ILH4QZ27CRQEMZYJDVHQSHYXK7HS6AYEMHMTTZ3ORKGJVY5QPWRQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.830966231 (617h49m22.702049267s) [noise=true , address=145.79.227.19:4079 , piece_id=XUZOEXYVA2W5V7CKBYLV6RJKMIT3HL3QH54THTYLDZGPP4AXIM4A, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.832084070 (617h49m22.703167115s) [noise=true , address=145.79.227.27:4690 , piece_id=TYGNBB2LNEGANSUW7I3YDAKLXK7CDIIB36SUJSOH77VW2VF5IVIQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.835498742 (617h49m22.706581777s) [noise=true , address=145.79.227.16:4911 , piece_id=PQOKWZ6NSQ7J2TZCRC6HLHASQGSGFCHNWEEVLX4U3FJOZ3ZE3ULA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.836971466 (617h49m22.708054501s) [noise=true , address=145.79.227.52:4447 , piece_id=5TINLO2QUFFV2UNMMY65BAQKQJZJ4HHX5MGSEJHA7DBV73KBDSMA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.844832134 (617h49m22.715915179s) [noise=true , address=145.79.227.51:4411 , piece_id=5SB3LW7HETIL2GTOFIIFYXJHOFNV4FFQX3AGGVMGJCRPUYHSCM4A, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.858135946 (617h49m22.729218991s) [noise=true , address=145.79.227.28:4725 , piece_id=RDLMTDF7IOPHLMBLXHPUG3Y5ZHLBJ4PORIOP7ULPXXUBB7XIUTIQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.908950354 (617h49m22.78003339s) [noise=true , address=145.79.227.32:4834 , piece_id=OEM2CWGUEEVM546YPRHPXPK3375OHGR4H5EDDYFJBMKP4KY5UOGQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.917357052 (617h49m22.788440097s) [noise=true , address=145.79.227.22:4511 , piece_id=JBK7Q3VEQTMXWKCTSYS64WZ6GWJ3B6P5CDTYHX4IHB5EHJQCSECA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:23.140118190 (617h49m23.011201245s) [noise=true , address=45.91.116.5:4299 , piece_id=ZI66OQ7OLIGLDP35Z3KD6PPDX6DU7ZA6I4FE7Y3HRBVGXQYGGOWQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:23.142880781 (617h49m23.013963826s) [noise=true , address=45.91.116.8:4046 , piece_id=23TJYJ2N55A4VIVDJRZRHTIQIZKEV5TXEIIBUITVOTNFDOGE5QTA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:23.143283688 (617h49m23.014366723s) [noise=true , address=45.91.116.10:4084 , piece_id=MV3HSJIRD3OLI7CSWDGTJWT6R446YKO5KVEPQKITEDHYLJ7F43FA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:23.143293416 (617h49m23.014376451s) [seg_pos={0 0}, upload=541411]: Segment reached optimal threshold of 13 pieces. Dec 31 09:18:23.143474071 (617h49m23.014557106s) [noise=true , address=45.91.116.9:4263 , piece_id=WEA4K2VUOA7437THKT62Q3JHKRBWOLZYX2AOIAI2ICLKCU5I7XIA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 09:18:23.143480193 (617h49m23.014563228s) [noise=true , address=45.91.116.9:4263 , piece_id=WEA4K2VUOA7437THKT62Q3JHKRBWOLZYX2AOIAI2ICLKCU5I7XIA, seg_pos={0 0}, upload=541411]: Long tail cancellation due to optimal threshold Dec 31 09:18:23.143761067 (617h49m23.014844122s) [noise=true , address=77.90.31.13:4627 , piece_id=GOCQCQVDWBMB4HOMBOG75DCU6C7ZMMZUNOMG6O4UJDHFODD5HVRA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 31 09:18:23.143770214 (617h49m23.01485325s) [noise=true , address=77.90.31.13:4627 , piece_id=GOCQCQVDWBMB4HOMBOG75DCU6C7ZMMZUNOMG6O4UJDHFODD5HVRA, seg_pos={0 0}, upload=541411]: Long tail cancellation due to optimal threshold Dec 31 09:18:23.143798368 (617h49m23.014881413s) [noise=true , address=45.91.116.7:4011 , piece_id=I27VVLYUQZPYAC7AHR6QWIZUQSSWU2JK364CLEIHKGPZBLCVRHJQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 09:18:23.143807746 (617h49m23.014890791s) [noise=true , address=45.91.116.7:4011 , piece_id=I27VVLYUQZPYAC7AHR6QWIZUQSSWU2JK364CLEIHKGPZBLCVRHJQ, seg_pos={0 0}, upload=541411]: Long tail cancellation due to optimal threshold Dec 31 09:18:23.143827183 (617h49m23.014910228s) [seg_pos={0 0}, upload=541411]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:18:23.143909510 (617h49m23.014992555s) [upload=541411]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:18:23.143916833 (617h49m23.014999878s) [upload=541411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:18:23.435627834 (617h49m23.306710879s) [upload=541411]: Done uploading segments... Dec 31 09:18:23.435651658 (617h49m23.306734703s) [upload=541411]: Upload finished. err: <nil> Duration: 6.248442909s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [......X...........X....X....XXX.XX.............XX...........] duration: 3.440045167s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:19:02 UTC) orbiter warning: context deadline exceeded Duration: 3.440122674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X....X...XXX..X.X.....X........XX....X..XX............X..X] duration: 30.003663765s trace: https://jaeger.dev.storj.io/trace/900c491f1cc7c8e FAIL (2025-12-31 09:18:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 09:18:20.587728627 (617h49m20.458811672s) [upload=541411]: Starting upload Dec 31 09:18:20.587804792 (617h49m20.458887827s) [upload=541411]: Done starting upload Dec 31 09:18:20.587874294 (617h49m20.458957329s) [upload=541411]: Uploading segments... Dec 31 09:18:20.588063024 (617h49m20.459146059s) [upload=541411]: Got next segment. Inline: false Dec 31 09:18:20.588122246 (617h49m20.459205291s) [upload=541411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:18:20.588128528 (617h49m20.459211573s) [upload=541411]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:18:20.837523231 (617h49m20.708606266s) [seg_pos={0 0}, upload=541411]: Begin upload segment... Dec 31 09:18:20.837588926 (617h49m20.708671971s) [seg_pos={0 0}, upload=541411]: Done begin upload segment. Dec 31 09:18:20.837636156 (617h49m20.708719191s) [noise=true , address=45.91.116.9:4263 , piece_id=WEA4K2VUOA7437THKT62Q3JHKRBWOLZYX2AOIAI2ICLKCU5I7XIA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.837643479 (617h49m20.708726515s) [noise=true , address=145.79.227.32:4834 , piece_id=OEM2CWGUEEVM546YPRHPXPK3375OHGR4H5EDDYFJBMKP4KY5UOGQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.838186404 (617h49m20.709269449s) [noise=true , address=45.91.116.10:4084 , piece_id=MV3HSJIRD3OLI7CSWDGTJWT6R446YKO5KVEPQKITEDHYLJ7F43FA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.838516232 (617h49m20.709599277s) [noise=true , address=45.91.116.7:4011 , piece_id=I27VVLYUQZPYAC7AHR6QWIZUQSSWU2JK364CLEIHKGPZBLCVRHJQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.838844076 (617h49m20.709927121s) [noise=true , address=145.79.227.27:4690 , piece_id=TYGNBB2LNEGANSUW7I3YDAKLXK7CDIIB36SUJSOH77VW2VF5IVIQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.839173824 (617h49m20.710256859s) [noise=true , address=145.79.227.46:4603 , piece_id=K7L7CU72XDU3NNUJN7EHOEK3Q6OYGEOWMC3KLT53SASDDNZEDS3A, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.839595557 (617h49m20.710678603s) [noise=true , address=45.91.116.8:4046 , piece_id=23TJYJ2N55A4VIVDJRZRHTIQIZKEV5TXEIIBUITVOTNFDOGE5QTA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.840020647 (617h49m20.711103682s) [noise=true , address=145.79.227.52:4447 , piece_id=5TINLO2QUFFV2UNMMY65BAQKQJZJ4HHX5MGSEJHA7DBV73KBDSMA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.840521220 (617h49m20.711604265s) [noise=true , address=45.91.116.5:4299 , piece_id=ZI66OQ7OLIGLDP35Z3KD6PPDX6DU7ZA6I4FE7Y3HRBVGXQYGGOWQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.840532150 (617h49m20.711615185s) [noise=true , address=145.79.227.19:4079 , piece_id=XUZOEXYVA2W5V7CKBYLV6RJKMIT3HL3QH54THTYLDZGPP4AXIM4A, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.840877878 (617h49m20.711960923s) [noise=true , address=145.79.227.16:4911 , piece_id=PQOKWZ6NSQ7J2TZCRC6HLHASQGSGFCHNWEEVLX4U3FJOZ3ZE3ULA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.840988410 (617h49m20.712071455s) [noise=true , address=145.79.227.22:4511 , piece_id=JBK7Q3VEQTMXWKCTSYS64WZ6GWJ3B6P5CDTYHX4IHB5EHJQCSECA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.841400774 (617h49m20.712483809s) [noise=true , address=145.79.227.9:4365 , piece_id=ILH4QZ27CRQEMZYJDVHQSHYXK7HS6AYEMHMTTZ3ORKGJVY5QPWRQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.841864347 (617h49m20.712947393s) [noise=true , address=145.79.227.28:4725 , piece_id=RDLMTDF7IOPHLMBLXHPUG3Y5ZHLBJ4PORIOP7ULPXXUBB7XIUTIQ, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.842369379 (617h49m20.713452414s) [noise=true , address=145.79.227.51:4411 , piece_id=5SB3LW7HETIL2GTOFIIFYXJHOFNV4FFQX3AGGVMGJCRPUYHSCM4A, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:20.842463137 (617h49m20.713546183s) [noise=true , address=77.90.31.13:4627 , piece_id=GOCQCQVDWBMB4HOMBOG75DCU6C7ZMMZUNOMG6O4UJDHFODD5HVRA, seg_pos={0 0}, upload=541411]: Uploading piece... Dec 31 09:18:22.539327069 (617h49m22.410410114s) [upload=541411]: Next returned nil segment Dec 31 09:18:22.539345043 (617h49m22.410428088s) [upload=541411]: Waiting for error group managing segments... Dec 31 09:18:22.827003025 (617h49m22.69808606s) [noise=true , address=145.79.227.46:4603 , piece_id=K7L7CU72XDU3NNUJN7EHOEK3Q6OYGEOWMC3KLT53SASDDNZEDS3A, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.827576277 (617h49m22.698659322s) [noise=true , address=145.79.227.9:4365 , piece_id=ILH4QZ27CRQEMZYJDVHQSHYXK7HS6AYEMHMTTZ3ORKGJVY5QPWRQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.830966231 (617h49m22.702049267s) [noise=true , address=145.79.227.19:4079 , piece_id=XUZOEXYVA2W5V7CKBYLV6RJKMIT3HL3QH54THTYLDZGPP4AXIM4A, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.832084070 (617h49m22.703167115s) [noise=true , address=145.79.227.27:4690 , piece_id=TYGNBB2LNEGANSUW7I3YDAKLXK7CDIIB36SUJSOH77VW2VF5IVIQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.835498742 (617h49m22.706581777s) [noise=true , address=145.79.227.16:4911 , piece_id=PQOKWZ6NSQ7J2TZCRC6HLHASQGSGFCHNWEEVLX4U3FJOZ3ZE3ULA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.836971466 (617h49m22.708054501s) [noise=true , address=145.79.227.52:4447 , piece_id=5TINLO2QUFFV2UNMMY65BAQKQJZJ4HHX5MGSEJHA7DBV73KBDSMA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.844832134 (617h49m22.715915179s) [noise=true , address=145.79.227.51:4411 , piece_id=5SB3LW7HETIL2GTOFIIFYXJHOFNV4FFQX3AGGVMGJCRPUYHSCM4A, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.858135946 (617h49m22.729218991s) [noise=true , address=145.79.227.28:4725 , piece_id=RDLMTDF7IOPHLMBLXHPUG3Y5ZHLBJ4PORIOP7ULPXXUBB7XIUTIQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.908950354 (617h49m22.78003339s) [noise=true , address=145.79.227.32:4834 , piece_id=OEM2CWGUEEVM546YPRHPXPK3375OHGR4H5EDDYFJBMKP4KY5UOGQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:22.917357052 (617h49m22.788440097s) [noise=true , address=145.79.227.22:4511 , piece_id=JBK7Q3VEQTMXWKCTSYS64WZ6GWJ3B6P5CDTYHX4IHB5EHJQCSECA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:23.140118190 (617h49m23.011201245s) [noise=true , address=45.91.116.5:4299 , piece_id=ZI66OQ7OLIGLDP35Z3KD6PPDX6DU7ZA6I4FE7Y3HRBVGXQYGGOWQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:23.142880781 (617h49m23.013963826s) [noise=true , address=45.91.116.8:4046 , piece_id=23TJYJ2N55A4VIVDJRZRHTIQIZKEV5TXEIIBUITVOTNFDOGE5QTA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:23.143283688 (617h49m23.014366723s) [noise=true , address=45.91.116.10:4084 , piece_id=MV3HSJIRD3OLI7CSWDGTJWT6R446YKO5KVEPQKITEDHYLJ7F43FA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: <nil> Dec 31 09:18:23.143293416 (617h49m23.014376451s) [seg_pos={0 0}, upload=541411]: Segment reached optimal threshold of 13 pieces. Dec 31 09:18:23.143474071 (617h49m23.014557106s) [noise=true , address=45.91.116.9:4263 , piece_id=WEA4K2VUOA7437THKT62Q3JHKRBWOLZYX2AOIAI2ICLKCU5I7XIA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 09:18:23.143480193 (617h49m23.014563228s) [noise=true , address=45.91.116.9:4263 , piece_id=WEA4K2VUOA7437THKT62Q3JHKRBWOLZYX2AOIAI2ICLKCU5I7XIA, seg_pos={0 0}, upload=541411]: Long tail cancellation due to optimal threshold Dec 31 09:18:23.143761067 (617h49m23.014844122s) [noise=true , address=77.90.31.13:4627 , piece_id=GOCQCQVDWBMB4HOMBOG75DCU6C7ZMMZUNOMG6O4UJDHFODD5HVRA, seg_pos={0 0}, upload=541411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 31 09:18:23.143770214 (617h49m23.01485325s) [noise=true , address=77.90.31.13:4627 , piece_id=GOCQCQVDWBMB4HOMBOG75DCU6C7ZMMZUNOMG6O4UJDHFODD5HVRA, seg_pos={0 0}, upload=541411]: Long tail cancellation due to optimal threshold Dec 31 09:18:23.143798368 (617h49m23.014881413s) [noise=true , address=45.91.116.7:4011 , piece_id=I27VVLYUQZPYAC7AHR6QWIZUQSSWU2JK364CLEIHKGPZBLCVRHJQ, seg_pos={0 0}, upload=541411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 09:18:23.143807746 (617h49m23.014890791s) [noise=true , address=45.91.116.7:4011 , piece_id=I27VVLYUQZPYAC7AHR6QWIZUQSSWU2JK364CLEIHKGPZBLCVRHJQ, seg_pos={0 0}, upload=541411]: Long tail cancellation due to optimal threshold Dec 31 09:18:23.143827183 (617h49m23.014910228s) [seg_pos={0 0}, upload=541411]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:18:23.143909510 (617h49m23.014992555s) [upload=541411]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:18:23.143916833 (617h49m23.014999878s) [upload=541411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:18:23.435627834 (617h49m23.306710879s) [upload=541411]: Done uploading segments... Dec 31 09:18:23.435651658 (617h49m23.306734703s) [upload=541411]: Upload finished. err: <nil> Duration: 30.003690116s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 24.991583812s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:18:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 24.991613288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.......X...........X....X....XXX.XX.............XX..........] duration: 3.537604725s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:17:57 UTC) orbiter warning: context deadline exceeded Duration: 3.537635914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.X....X...XXX..X.X.....X........XX....X..XX............X..] duration: 6.054348033s trace: https://jaeger.dev.storj.io/trace/187fc53e91d7117f FAIL (2025-12-31 09:17:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WWQ66222W5Y6V42K53D7CD3WU6BQB776SRKPMEF7QHBR2HP3I36Q) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: GBHG3S3MRC6XBG52E22ZHC4CA4ILWXKRBR7HHERJFGMFWYYTJWZA) context deadline exceeded ======LOGS====== segments... Dec 31 09:16:28.809972708 (617h47m28.681055743s) [upload=541383]: Got next segment. Inline: false Dec 31 09:16:28.810013496 (617h47m28.681096531s) [upload=541383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:16:28.810018746 (617h47m28.681101781s) [upload=541383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:16:29.057169666 (617h47m28.928252712s) [seg_pos={0 0}, upload=541383]: Begin upload segment... Dec 31 09:16:29.057234290 (617h47m28.928317325s) [seg_pos={0 0}, upload=541383]: Done begin upload segment. Dec 31 09:16:29.057287911 (617h47m28.928370956s) [noise=true , address=145.79.227.36:4517 , piece_id=DBKXBJK6UOLUCSUB4WMJGVY6XDL2PULSJ7ILKPC5C3BAIPRS3JYQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.057314352 (617h47m28.928397387s) [noise=true , address=145.79.227.51:4411 , piece_id=JWTXWFX2TYY2GDY3ZJ7MDZZ2QI6TWMCUBQZHR3W473JG22NXVK2A, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.057945113 (617h47m28.929028158s) [noise=true , address=145.79.227.24:4581 , piece_id=AODJ2J7U5E2ZAFBDFFAAPRRKNVY64AUYNFBKYL5CSH7VKPWMQXJA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.058087674 (617h47m28.929170709s) [noise=true , address=45.91.116.5:4300 , piece_id=WWQ66222W5Y6V42K53D7CD3WU6BQB776SRKPMEF7QHBR2HP3I36Q, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.058563050 (617h47m28.929646095s) [noise=true , address=145.79.227.35:4482 , piece_id=SLB7NZPSROK5SXUNCZKAQPZ6ACK22HWIG4RTOJSD3ZWJUOA4X2SQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.058942122 (617h47m28.930025167s) [noise=true , address=45.91.116.4:4939 , piece_id=QUQAXRS54HU7GYEDBINNBMHAB5OM3W24FGZXO322LLV6O6TJTC5A, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.059268172 (617h47m28.930351207s) [noise=true , address=77.90.31.21:4878 , piece_id=GBHG3S3MRC6XBG52E22ZHC4CA4ILWXKRBR7HHERJFGMFWYYTJWZA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.059564216 (617h47m28.930647251s) [noise=true , address=145.79.227.45:4231 , piece_id=LZDFUQTP3S4M23CQ3JDD2CA7K6HS72JAYLPCDWFSFUFOII73MEWQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.059837627 (617h47m28.930920672s) [noise=true , address=145.79.227.34:4906 , piece_id=5DDAON5NPOTE4D3Z4JO3AFNFVFYKZPC4ABYLZFJ5M7LLCKGSYQDA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.060136846 (617h47m28.931219881s) [noise=true , address=45.91.116.7:4010 , piece_id=WJT4PYU5IYFEPBOROREV3EPO6GEGBBV6ZGJWOVX4ZLXWMM6KLFIQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.060562957 (617h47m28.931646002s) [noise=true , address=145.79.227.15:4874 , piece_id=R5V5ZIFD2KVVQGSDTS4JGYZXM2PGSZRSJ3KCF63MYN2DIUBZ2FYA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.060873048 (617h47m28.931956093s) [noise=true , address=45.91.116.9:4261 , piece_id=ICVTBBWHMHZMLCBBFJ6JYTPOJ5UEBOKMPQEKR3VFP3BYSCOSRLOA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.062759761 (617h47m28.933842796s) [noise=true , address=145.79.227.12:4767 , piece_id=BE5KSZ4BITY5RLKUN54QS2ZRVAAZ4C7AWBM2MJ5ZFTPQJBZUQ6FQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.064311105 (617h47m28.93539414s) [noise=true , address=145.79.227.5:4222 , piece_id=5JRNR35RLDJ57BZEKIZCRE2YPIBO7K2WNKHOIK6UBY6TLG25UIJA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.065749213 (617h47m28.936832248s) [noise=true , address=45.91.116.6:4975 , piece_id=6GYI4ASX2C5LO74IEVZTSJC4ZHPS6OMQGVCS6BVAVXFFC2GTR4NA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.066909834 (617h47m28.937992869s) [noise=true , address=45.91.116.8:4047 , piece_id=TJFXK3O7U5ZKRDYXJXC2REXWZZWOOVRTEQT2BQUVHGNQ6XPQ7AMQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:30.749020683 (617h47m30.620103718s) [upload=541383]: Next returned nil segment Dec 31 09:16:30.749033587 (617h47m30.620116622s) [upload=541383]: Waiting for error group managing segments... Dec 31 09:16:31.044924225 (617h47m30.91600726s) [noise=true , address=145.79.227.34:4906 , piece_id=5DDAON5NPOTE4D3Z4JO3AFNFVFYKZPC4ABYLZFJ5M7LLCKGSYQDA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.052332632 (617h47m30.923415667s) [noise=true , address=145.79.227.51:4411 , piece_id=JWTXWFX2TYY2GDY3ZJ7MDZZ2QI6TWMCUBQZHR3W473JG22NXVK2A, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.058386960 (617h47m30.929469995s) [noise=true , address=145.79.227.36:4517 , piece_id=DBKXBJK6UOLUCSUB4WMJGVY6XDL2PULSJ7ILKPC5C3BAIPRS3JYQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.058839141 (617h47m30.929922176s) [noise=true , address=145.79.227.24:4581 , piece_id=AODJ2J7U5E2ZAFBDFFAAPRRKNVY64AUYNFBKYL5CSH7VKPWMQXJA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.074907167 (617h47m30.945990212s) [noise=true , address=145.79.227.15:4874 , piece_id=R5V5ZIFD2KVVQGSDTS4JGYZXM2PGSZRSJ3KCF63MYN2DIUBZ2FYA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.077326694 (617h47m30.948409739s) [noise=true , address=145.79.227.35:4482 , piece_id=SLB7NZPSROK5SXUNCZKAQPZ6ACK22HWIG4RTOJSD3ZWJUOA4X2SQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.079750429 (617h47m30.950833464s) [noise=true , address=145.79.227.45:4231 , piece_id=LZDFUQTP3S4M23CQ3JDD2CA7K6HS72JAYLPCDWFSFUFOII73MEWQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.080173144 (617h47m30.951256189s) [noise=true , address=145.79.227.12:4767 , piece_id=BE5KSZ4BITY5RLKUN54QS2ZRVAAZ4C7AWBM2MJ5ZFTPQJBZUQ6FQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.123631856 (617h47m30.994714891s) [noise=true , address=145.79.227.5:4222 , piece_id=5JRNR35RLDJ57BZEKIZCRE2YPIBO7K2WNKHOIK6UBY6TLG25UIJA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.250858594 (617h47m31.121941629s) [noise=true , address=77.90.31.21:4878 , piece_id=GBHG3S3MRC6XBG52E22ZHC4CA4ILWXKRBR7HHERJFGMFWYYTJWZA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.383105852 (617h47m31.254188897s) [noise=true , address=45.91.116.6:4975 , piece_id=6GYI4ASX2C5LO74IEVZTSJC4ZHPS6OMQGVCS6BVAVXFFC2GTR4NA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.402680825 (617h47m31.27376386s) [noise=true , address=45.91.116.9:4261 , piece_id=ICVTBBWHMHZMLCBBFJ6JYTPOJ5UEBOKMPQEKR3VFP3BYSCOSRLOA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.465570887 (617h47m31.336653932s) [noise=true , address=45.91.116.5:4300 , piece_id=WWQ66222W5Y6V42K53D7CD3WU6BQB776SRKPMEF7QHBR2HP3I36Q, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.465594813 (617h47m31.336677858s) [seg_pos={0 0}, upload=541383]: Segment reached optimal threshold of 13 pieces. Dec 31 09:16:31.465793280 (617h47m31.336876325s) [noise=true , address=45.91.116.8:4047 , piece_id=TJFXK3O7U5ZKRDYXJXC2REXWZZWOOVRTEQT2BQUVHGNQ6XPQ7AMQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 31 09:16:31.465848555 (617h47m31.336931611s) [noise=true , address=45.91.116.8:4047 , piece_id=TJFXK3O7U5ZKRDYXJXC2REXWZZWOOVRTEQT2BQUVHGNQ6XPQ7AMQ, seg_pos={0 0}, upload=541383]: Long tail cancellation due to optimal threshold Dec 31 09:16:31.465878142 (617h47m31.336961187s) [noise=true , address=45.91.116.7:4010 , piece_id=WJT4PYU5IYFEPBOROREV3EPO6GEGBBV6ZGJWOVX4ZLXWMM6KLFIQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:52510->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:52510->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52510->45.91.116.7:4010: use of closed network connection Dec 31 09:16:31.465888101 (617h47m31.336971146s) [noise=true , address=45.91.116.7:4010 , piece_id=WJT4PYU5IYFEPBOROREV3EPO6GEGBBV6ZGJWOVX4ZLXWMM6KLFIQ, seg_pos={0 0}, upload=541383]: Long tail cancellation due to optimal threshold Dec 31 09:16:31.466007848 (617h47m31.337090883s) [noise=true , address=45.91.116.4:4939 , piece_id=QUQAXRS54HU7GYEDBINNBMHAB5OM3W24FGZXO322LLV6O6TJTC5A, seg_pos={0 0}, upload=541383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:39962->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:39962->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39962->45.91.116.4:4939: use of closed network connection Dec 31 09:16:31.466014151 (617h47m31.337097186s) [noise=true , address=45.91.116.4:4939 , piece_id=QUQAXRS54HU7GYEDBINNBMHAB5OM3W24FGZXO322LLV6O6TJTC5A, seg_pos={0 0}, upload=541383]: Long tail cancellation due to optimal threshold Dec 31 09:16:31.466036803 (617h47m31.337119849s) [seg_pos={0 0}, upload=541383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:16:31.466123329 (617h47m31.337206374s) [upload=541383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:16:31.466131234 (617h47m31.337214269s) [upload=541383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:16:31.754756697 (617h47m31.625839742s) [upload=541383]: Done uploading segments... Dec 31 09:16:31.754776014 (617h47m31.625859059s) [upload=541383]: Upload finished. err: <nil> Duration: 6.054377069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 17.815596678s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:17:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 17.815618729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.X....X...XXX..X.X.....X........XX....X..XX............X.] duration: 9.006468744s trace: https://jaeger.dev.storj.io/trace/187fc53e91d7117f FAIL (2025-12-31 09:17:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WWQ66222W5Y6V42K53D7CD3WU6BQB776SRKPMEF7QHBR2HP3I36Q) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: GBHG3S3MRC6XBG52E22ZHC4CA4ILWXKRBR7HHERJFGMFWYYTJWZA) context deadline exceeded ======LOGS====== segments... Dec 31 09:16:28.809972708 (617h47m28.681055743s) [upload=541383]: Got next segment. Inline: false Dec 31 09:16:28.810013496 (617h47m28.681096531s) [upload=541383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:16:28.810018746 (617h47m28.681101781s) [upload=541383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:16:29.057169666 (617h47m28.928252712s) [seg_pos={0 0}, upload=541383]: Begin upload segment... Dec 31 09:16:29.057234290 (617h47m28.928317325s) [seg_pos={0 0}, upload=541383]: Done begin upload segment. Dec 31 09:16:29.057287911 (617h47m28.928370956s) [noise=true , address=145.79.227.36:4517 , piece_id=DBKXBJK6UOLUCSUB4WMJGVY6XDL2PULSJ7ILKPC5C3BAIPRS3JYQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.057314352 (617h47m28.928397387s) [noise=true , address=145.79.227.51:4411 , piece_id=JWTXWFX2TYY2GDY3ZJ7MDZZ2QI6TWMCUBQZHR3W473JG22NXVK2A, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.057945113 (617h47m28.929028158s) [noise=true , address=145.79.227.24:4581 , piece_id=AODJ2J7U5E2ZAFBDFFAAPRRKNVY64AUYNFBKYL5CSH7VKPWMQXJA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.058087674 (617h47m28.929170709s) [noise=true , address=45.91.116.5:4300 , piece_id=WWQ66222W5Y6V42K53D7CD3WU6BQB776SRKPMEF7QHBR2HP3I36Q, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.058563050 (617h47m28.929646095s) [noise=true , address=145.79.227.35:4482 , piece_id=SLB7NZPSROK5SXUNCZKAQPZ6ACK22HWIG4RTOJSD3ZWJUOA4X2SQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.058942122 (617h47m28.930025167s) [noise=true , address=45.91.116.4:4939 , piece_id=QUQAXRS54HU7GYEDBINNBMHAB5OM3W24FGZXO322LLV6O6TJTC5A, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.059268172 (617h47m28.930351207s) [noise=true , address=77.90.31.21:4878 , piece_id=GBHG3S3MRC6XBG52E22ZHC4CA4ILWXKRBR7HHERJFGMFWYYTJWZA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.059564216 (617h47m28.930647251s) [noise=true , address=145.79.227.45:4231 , piece_id=LZDFUQTP3S4M23CQ3JDD2CA7K6HS72JAYLPCDWFSFUFOII73MEWQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.059837627 (617h47m28.930920672s) [noise=true , address=145.79.227.34:4906 , piece_id=5DDAON5NPOTE4D3Z4JO3AFNFVFYKZPC4ABYLZFJ5M7LLCKGSYQDA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.060136846 (617h47m28.931219881s) [noise=true , address=45.91.116.7:4010 , piece_id=WJT4PYU5IYFEPBOROREV3EPO6GEGBBV6ZGJWOVX4ZLXWMM6KLFIQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.060562957 (617h47m28.931646002s) [noise=true , address=145.79.227.15:4874 , piece_id=R5V5ZIFD2KVVQGSDTS4JGYZXM2PGSZRSJ3KCF63MYN2DIUBZ2FYA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.060873048 (617h47m28.931956093s) [noise=true , address=45.91.116.9:4261 , piece_id=ICVTBBWHMHZMLCBBFJ6JYTPOJ5UEBOKMPQEKR3VFP3BYSCOSRLOA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.062759761 (617h47m28.933842796s) [noise=true , address=145.79.227.12:4767 , piece_id=BE5KSZ4BITY5RLKUN54QS2ZRVAAZ4C7AWBM2MJ5ZFTPQJBZUQ6FQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.064311105 (617h47m28.93539414s) [noise=true , address=145.79.227.5:4222 , piece_id=5JRNR35RLDJ57BZEKIZCRE2YPIBO7K2WNKHOIK6UBY6TLG25UIJA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.065749213 (617h47m28.936832248s) [noise=true , address=45.91.116.6:4975 , piece_id=6GYI4ASX2C5LO74IEVZTSJC4ZHPS6OMQGVCS6BVAVXFFC2GTR4NA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.066909834 (617h47m28.937992869s) [noise=true , address=45.91.116.8:4047 , piece_id=TJFXK3O7U5ZKRDYXJXC2REXWZZWOOVRTEQT2BQUVHGNQ6XPQ7AMQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:30.749020683 (617h47m30.620103718s) [upload=541383]: Next returned nil segment Dec 31 09:16:30.749033587 (617h47m30.620116622s) [upload=541383]: Waiting for error group managing segments... Dec 31 09:16:31.044924225 (617h47m30.91600726s) [noise=true , address=145.79.227.34:4906 , piece_id=5DDAON5NPOTE4D3Z4JO3AFNFVFYKZPC4ABYLZFJ5M7LLCKGSYQDA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.052332632 (617h47m30.923415667s) [noise=true , address=145.79.227.51:4411 , piece_id=JWTXWFX2TYY2GDY3ZJ7MDZZ2QI6TWMCUBQZHR3W473JG22NXVK2A, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.058386960 (617h47m30.929469995s) [noise=true , address=145.79.227.36:4517 , piece_id=DBKXBJK6UOLUCSUB4WMJGVY6XDL2PULSJ7ILKPC5C3BAIPRS3JYQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.058839141 (617h47m30.929922176s) [noise=true , address=145.79.227.24:4581 , piece_id=AODJ2J7U5E2ZAFBDFFAAPRRKNVY64AUYNFBKYL5CSH7VKPWMQXJA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.074907167 (617h47m30.945990212s) [noise=true , address=145.79.227.15:4874 , piece_id=R5V5ZIFD2KVVQGSDTS4JGYZXM2PGSZRSJ3KCF63MYN2DIUBZ2FYA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.077326694 (617h47m30.948409739s) [noise=true , address=145.79.227.35:4482 , piece_id=SLB7NZPSROK5SXUNCZKAQPZ6ACK22HWIG4RTOJSD3ZWJUOA4X2SQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.079750429 (617h47m30.950833464s) [noise=true , address=145.79.227.45:4231 , piece_id=LZDFUQTP3S4M23CQ3JDD2CA7K6HS72JAYLPCDWFSFUFOII73MEWQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.080173144 (617h47m30.951256189s) [noise=true , address=145.79.227.12:4767 , piece_id=BE5KSZ4BITY5RLKUN54QS2ZRVAAZ4C7AWBM2MJ5ZFTPQJBZUQ6FQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.123631856 (617h47m30.994714891s) [noise=true , address=145.79.227.5:4222 , piece_id=5JRNR35RLDJ57BZEKIZCRE2YPIBO7K2WNKHOIK6UBY6TLG25UIJA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.250858594 (617h47m31.121941629s) [noise=true , address=77.90.31.21:4878 , piece_id=GBHG3S3MRC6XBG52E22ZHC4CA4ILWXKRBR7HHERJFGMFWYYTJWZA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.383105852 (617h47m31.254188897s) [noise=true , address=45.91.116.6:4975 , piece_id=6GYI4ASX2C5LO74IEVZTSJC4ZHPS6OMQGVCS6BVAVXFFC2GTR4NA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.402680825 (617h47m31.27376386s) [noise=true , address=45.91.116.9:4261 , piece_id=ICVTBBWHMHZMLCBBFJ6JYTPOJ5UEBOKMPQEKR3VFP3BYSCOSRLOA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.465570887 (617h47m31.336653932s) [noise=true , address=45.91.116.5:4300 , piece_id=WWQ66222W5Y6V42K53D7CD3WU6BQB776SRKPMEF7QHBR2HP3I36Q, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.465594813 (617h47m31.336677858s) [seg_pos={0 0}, upload=541383]: Segment reached optimal threshold of 13 pieces. Dec 31 09:16:31.465793280 (617h47m31.336876325s) [noise=true , address=45.91.116.8:4047 , piece_id=TJFXK3O7U5ZKRDYXJXC2REXWZZWOOVRTEQT2BQUVHGNQ6XPQ7AMQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 31 09:16:31.465848555 (617h47m31.336931611s) [noise=true , address=45.91.116.8:4047 , piece_id=TJFXK3O7U5ZKRDYXJXC2REXWZZWOOVRTEQT2BQUVHGNQ6XPQ7AMQ, seg_pos={0 0}, upload=541383]: Long tail cancellation due to optimal threshold Dec 31 09:16:31.465878142 (617h47m31.336961187s) [noise=true , address=45.91.116.7:4010 , piece_id=WJT4PYU5IYFEPBOROREV3EPO6GEGBBV6ZGJWOVX4ZLXWMM6KLFIQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:52510->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:52510->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52510->45.91.116.7:4010: use of closed network connection Dec 31 09:16:31.465888101 (617h47m31.336971146s) [noise=true , address=45.91.116.7:4010 , piece_id=WJT4PYU5IYFEPBOROREV3EPO6GEGBBV6ZGJWOVX4ZLXWMM6KLFIQ, seg_pos={0 0}, upload=541383]: Long tail cancellation due to optimal threshold Dec 31 09:16:31.466007848 (617h47m31.337090883s) [noise=true , address=45.91.116.4:4939 , piece_id=QUQAXRS54HU7GYEDBINNBMHAB5OM3W24FGZXO322LLV6O6TJTC5A, seg_pos={0 0}, upload=541383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:39962->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:39962->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39962->45.91.116.4:4939: use of closed network connection Dec 31 09:16:31.466014151 (617h47m31.337097186s) [noise=true , address=45.91.116.4:4939 , piece_id=QUQAXRS54HU7GYEDBINNBMHAB5OM3W24FGZXO322LLV6O6TJTC5A, seg_pos={0 0}, upload=541383]: Long tail cancellation due to optimal threshold Dec 31 09:16:31.466036803 (617h47m31.337119849s) [seg_pos={0 0}, upload=541383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:16:31.466123329 (617h47m31.337206374s) [upload=541383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:16:31.466131234 (617h47m31.337214269s) [upload=541383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:16:31.754756697 (617h47m31.625839742s) [upload=541383]: Done uploading segments... Dec 31 09:16:31.754776014 (617h47m31.625859059s) [upload=541383]: Upload finished. err: <nil> Duration: 9.006504252s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 37.02171101s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:17:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 37.021741818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X.X....X...XXX..X.X.....X........XX....X..XX............X] duration: 30.002938219s trace: https://jaeger.dev.storj.io/trace/187fc53e91d7117f FAIL (2025-12-31 09:16:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: WWQ66222W5Y6V42K53D7CD3WU6BQB776SRKPMEF7QHBR2HP3I36Q) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: GBHG3S3MRC6XBG52E22ZHC4CA4ILWXKRBR7HHERJFGMFWYYTJWZA) context deadline exceeded ======LOGS====== segments... Dec 31 09:16:28.809972708 (617h47m28.681055743s) [upload=541383]: Got next segment. Inline: false Dec 31 09:16:28.810013496 (617h47m28.681096531s) [upload=541383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:16:28.810018746 (617h47m28.681101781s) [upload=541383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:16:29.057169666 (617h47m28.928252712s) [seg_pos={0 0}, upload=541383]: Begin upload segment... Dec 31 09:16:29.057234290 (617h47m28.928317325s) [seg_pos={0 0}, upload=541383]: Done begin upload segment. Dec 31 09:16:29.057287911 (617h47m28.928370956s) [noise=true , address=145.79.227.36:4517 , piece_id=DBKXBJK6UOLUCSUB4WMJGVY6XDL2PULSJ7ILKPC5C3BAIPRS3JYQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.057314352 (617h47m28.928397387s) [noise=true , address=145.79.227.51:4411 , piece_id=JWTXWFX2TYY2GDY3ZJ7MDZZ2QI6TWMCUBQZHR3W473JG22NXVK2A, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.057945113 (617h47m28.929028158s) [noise=true , address=145.79.227.24:4581 , piece_id=AODJ2J7U5E2ZAFBDFFAAPRRKNVY64AUYNFBKYL5CSH7VKPWMQXJA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.058087674 (617h47m28.929170709s) [noise=true , address=45.91.116.5:4300 , piece_id=WWQ66222W5Y6V42K53D7CD3WU6BQB776SRKPMEF7QHBR2HP3I36Q, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.058563050 (617h47m28.929646095s) [noise=true , address=145.79.227.35:4482 , piece_id=SLB7NZPSROK5SXUNCZKAQPZ6ACK22HWIG4RTOJSD3ZWJUOA4X2SQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.058942122 (617h47m28.930025167s) [noise=true , address=45.91.116.4:4939 , piece_id=QUQAXRS54HU7GYEDBINNBMHAB5OM3W24FGZXO322LLV6O6TJTC5A, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.059268172 (617h47m28.930351207s) [noise=true , address=77.90.31.21:4878 , piece_id=GBHG3S3MRC6XBG52E22ZHC4CA4ILWXKRBR7HHERJFGMFWYYTJWZA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.059564216 (617h47m28.930647251s) [noise=true , address=145.79.227.45:4231 , piece_id=LZDFUQTP3S4M23CQ3JDD2CA7K6HS72JAYLPCDWFSFUFOII73MEWQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.059837627 (617h47m28.930920672s) [noise=true , address=145.79.227.34:4906 , piece_id=5DDAON5NPOTE4D3Z4JO3AFNFVFYKZPC4ABYLZFJ5M7LLCKGSYQDA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.060136846 (617h47m28.931219881s) [noise=true , address=45.91.116.7:4010 , piece_id=WJT4PYU5IYFEPBOROREV3EPO6GEGBBV6ZGJWOVX4ZLXWMM6KLFIQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.060562957 (617h47m28.931646002s) [noise=true , address=145.79.227.15:4874 , piece_id=R5V5ZIFD2KVVQGSDTS4JGYZXM2PGSZRSJ3KCF63MYN2DIUBZ2FYA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.060873048 (617h47m28.931956093s) [noise=true , address=45.91.116.9:4261 , piece_id=ICVTBBWHMHZMLCBBFJ6JYTPOJ5UEBOKMPQEKR3VFP3BYSCOSRLOA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.062759761 (617h47m28.933842796s) [noise=true , address=145.79.227.12:4767 , piece_id=BE5KSZ4BITY5RLKUN54QS2ZRVAAZ4C7AWBM2MJ5ZFTPQJBZUQ6FQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.064311105 (617h47m28.93539414s) [noise=true , address=145.79.227.5:4222 , piece_id=5JRNR35RLDJ57BZEKIZCRE2YPIBO7K2WNKHOIK6UBY6TLG25UIJA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.065749213 (617h47m28.936832248s) [noise=true , address=45.91.116.6:4975 , piece_id=6GYI4ASX2C5LO74IEVZTSJC4ZHPS6OMQGVCS6BVAVXFFC2GTR4NA, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:29.066909834 (617h47m28.937992869s) [noise=true , address=45.91.116.8:4047 , piece_id=TJFXK3O7U5ZKRDYXJXC2REXWZZWOOVRTEQT2BQUVHGNQ6XPQ7AMQ, seg_pos={0 0}, upload=541383]: Uploading piece... Dec 31 09:16:30.749020683 (617h47m30.620103718s) [upload=541383]: Next returned nil segment Dec 31 09:16:30.749033587 (617h47m30.620116622s) [upload=541383]: Waiting for error group managing segments... Dec 31 09:16:31.044924225 (617h47m30.91600726s) [noise=true , address=145.79.227.34:4906 , piece_id=5DDAON5NPOTE4D3Z4JO3AFNFVFYKZPC4ABYLZFJ5M7LLCKGSYQDA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.052332632 (617h47m30.923415667s) [noise=true , address=145.79.227.51:4411 , piece_id=JWTXWFX2TYY2GDY3ZJ7MDZZ2QI6TWMCUBQZHR3W473JG22NXVK2A, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.058386960 (617h47m30.929469995s) [noise=true , address=145.79.227.36:4517 , piece_id=DBKXBJK6UOLUCSUB4WMJGVY6XDL2PULSJ7ILKPC5C3BAIPRS3JYQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.058839141 (617h47m30.929922176s) [noise=true , address=145.79.227.24:4581 , piece_id=AODJ2J7U5E2ZAFBDFFAAPRRKNVY64AUYNFBKYL5CSH7VKPWMQXJA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.074907167 (617h47m30.945990212s) [noise=true , address=145.79.227.15:4874 , piece_id=R5V5ZIFD2KVVQGSDTS4JGYZXM2PGSZRSJ3KCF63MYN2DIUBZ2FYA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.077326694 (617h47m30.948409739s) [noise=true , address=145.79.227.35:4482 , piece_id=SLB7NZPSROK5SXUNCZKAQPZ6ACK22HWIG4RTOJSD3ZWJUOA4X2SQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.079750429 (617h47m30.950833464s) [noise=true , address=145.79.227.45:4231 , piece_id=LZDFUQTP3S4M23CQ3JDD2CA7K6HS72JAYLPCDWFSFUFOII73MEWQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.080173144 (617h47m30.951256189s) [noise=true , address=145.79.227.12:4767 , piece_id=BE5KSZ4BITY5RLKUN54QS2ZRVAAZ4C7AWBM2MJ5ZFTPQJBZUQ6FQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.123631856 (617h47m30.994714891s) [noise=true , address=145.79.227.5:4222 , piece_id=5JRNR35RLDJ57BZEKIZCRE2YPIBO7K2WNKHOIK6UBY6TLG25UIJA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.250858594 (617h47m31.121941629s) [noise=true , address=77.90.31.21:4878 , piece_id=GBHG3S3MRC6XBG52E22ZHC4CA4ILWXKRBR7HHERJFGMFWYYTJWZA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.383105852 (617h47m31.254188897s) [noise=true , address=45.91.116.6:4975 , piece_id=6GYI4ASX2C5LO74IEVZTSJC4ZHPS6OMQGVCS6BVAVXFFC2GTR4NA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.402680825 (617h47m31.27376386s) [noise=true , address=45.91.116.9:4261 , piece_id=ICVTBBWHMHZMLCBBFJ6JYTPOJ5UEBOKMPQEKR3VFP3BYSCOSRLOA, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.465570887 (617h47m31.336653932s) [noise=true , address=45.91.116.5:4300 , piece_id=WWQ66222W5Y6V42K53D7CD3WU6BQB776SRKPMEF7QHBR2HP3I36Q, seg_pos={0 0}, upload=541383]: Done uploading piece. err: <nil> Dec 31 09:16:31.465594813 (617h47m31.336677858s) [seg_pos={0 0}, upload=541383]: Segment reached optimal threshold of 13 pieces. Dec 31 09:16:31.465793280 (617h47m31.336876325s) [noise=true , address=45.91.116.8:4047 , piece_id=TJFXK3O7U5ZKRDYXJXC2REXWZZWOOVRTEQT2BQUVHGNQ6XPQ7AMQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 31 09:16:31.465848555 (617h47m31.336931611s) [noise=true , address=45.91.116.8:4047 , piece_id=TJFXK3O7U5ZKRDYXJXC2REXWZZWOOVRTEQT2BQUVHGNQ6XPQ7AMQ, seg_pos={0 0}, upload=541383]: Long tail cancellation due to optimal threshold Dec 31 09:16:31.465878142 (617h47m31.336961187s) [noise=true , address=45.91.116.7:4010 , piece_id=WJT4PYU5IYFEPBOROREV3EPO6GEGBBV6ZGJWOVX4ZLXWMM6KLFIQ, seg_pos={0 0}, upload=541383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:52510->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:52510->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52510->45.91.116.7:4010: use of closed network connection Dec 31 09:16:31.465888101 (617h47m31.336971146s) [noise=true , address=45.91.116.7:4010 , piece_id=WJT4PYU5IYFEPBOROREV3EPO6GEGBBV6ZGJWOVX4ZLXWMM6KLFIQ, seg_pos={0 0}, upload=541383]: Long tail cancellation due to optimal threshold Dec 31 09:16:31.466007848 (617h47m31.337090883s) [noise=true , address=45.91.116.4:4939 , piece_id=QUQAXRS54HU7GYEDBINNBMHAB5OM3W24FGZXO322LLV6O6TJTC5A, seg_pos={0 0}, upload=541383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:39962->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:39962->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39962->45.91.116.4:4939: use of closed network connection Dec 31 09:16:31.466014151 (617h47m31.337097186s) [noise=true , address=45.91.116.4:4939 , piece_id=QUQAXRS54HU7GYEDBINNBMHAB5OM3W24FGZXO322LLV6O6TJTC5A, seg_pos={0 0}, upload=541383]: Long tail cancellation due to optimal threshold Dec 31 09:16:31.466036803 (617h47m31.337119849s) [seg_pos={0 0}, upload=541383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:16:31.466123329 (617h47m31.337206374s) [upload=541383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:16:31.466131234 (617h47m31.337214269s) [upload=541383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:16:31.754756697 (617h47m31.625839742s) [upload=541383]: Done uploading segments... Dec 31 09:16:31.754776014 (617h47m31.625859059s) [upload=541383]: Upload finished. err: <nil> Duration: 30.002951453s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [........X...........X....X....XXX.XX.............XX.........] duration: 3.514382166s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:16:49 UTC) orbiter warning: context deadline exceeded Duration: 3.514409727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.........X...........X....X....XXX.XX.............XX........] duration: 3.396589721s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:16:08 UTC) orbiter warning: context deadline exceeded Duration: 3.396624347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 33.336574334s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:16:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 33.336593421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X.X....X...XXX..X.X.....X........XX....X..XX............] duration: 7.559335212s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:15:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 7.559363346s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..........X...........X....X....XXX.XX.............XX.......] duration: 3.390218076s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:15:26 UTC) orbiter warning: context deadline exceeded Duration: 3.390287429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 15.377891347s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:15:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 15.377911094s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 40.343117342s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:14:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 40.343137601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X....X.X....X...XXX..X.X.....X........XX....X..XX...........] duration: 5.953274565s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:14:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 5.953292599s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...........X...........X....X....XXX.XX.............XX......] duration: 3.535484995s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:14:20 UTC) orbiter warning: context deadline exceeded Duration: 3.535514472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 31.844900721s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:14:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 31.844921801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X....X.X....X...XXX..X.X.....X........XX....X..XX..........] duration: 7.74085258s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:14:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 7.740883288s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [............X...........X....X....XXX.XX.............XX.....] duration: 3.473391719s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:13:55 UTC) orbiter warning: context deadline exceeded Duration: 3.473415887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 26.187789814s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:13:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 26.187818328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X....X.X....X...XXX..X.X.....X........XX....X..XX.........] duration: 5.807822145s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:13:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 5.80784129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 18.927945932s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:13:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 18.927986218s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.............X...........X....X....XXX.XX.............XX....] duration: 4.117603648s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:12:54 UTC) orbiter warning: context deadline exceeded Duration: 4.117628256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X....X.X....X...XXX..X.X.....X........XX....X..XX........] duration: 5.888257363s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:12:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 5.888287461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 35.569902748s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:12:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 35.56992007s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..............X...........X....X....XXX.XX.............XX...] duration: 3.742654472s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:11:52 UTC) orbiter warning: context deadline exceeded Duration: 3.742678458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X....X.X....X...XXX..X.X.....X........XX....X..XX.......] duration: 5.991809644s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:11:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 5.991894354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 15.656168388s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:10:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 15.656194557s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X....X.X....X...XXX..X.X.....X........XX....X..XX......] duration: 16.163933781s trace: https://jaeger.dev.storj.io/trace/354579860c475801 FAIL (2025-12-31 09:10:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA) context deadline exceeded ======LOGS====== Dec 31 09:06:05.623627963 (617h37m5.494710998s) [upload=541230]: Starting upload Dec 31 09:06:05.623687576 (617h37m5.494770611s) [upload=541230]: Done starting upload Dec 31 09:06:05.623797315 (617h37m5.49488035s) [upload=541230]: Uploading segments... Dec 31 09:06:05.623963341 (617h37m5.495046377s) [upload=541230]: Got next segment. Inline: false Dec 31 09:06:05.624003378 (617h37m5.495086423s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 09:06:05.624008136 (617h37m5.495091172s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 09:06:05.867160403 (617h37m5.738243438s) [seg_pos={0 0}, upload=541230]: Begin upload segment... Dec 31 09:06:05.867217431 (617h37m5.738300466s) [seg_pos={0 0}, upload=541230]: Done begin upload segment. Dec 31 09:06:05.867253921 (617h37m5.738336956s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867292885 (617h37m5.73837592s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.867729256 (617h37m5.738812301s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868133556 (617h37m5.739216591s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.868533548 (617h37m5.739616593s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869018351 (617h37m5.740101386s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869464290 (617h37m5.740547325s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.869870603 (617h37m5.740953638s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870347171 (617h37m5.741430216s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.870799663 (617h37m5.741882698s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871259889 (617h37m5.742342924s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.871817150 (617h37m5.742900195s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872641620 (617h37m5.743724665s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.872695763 (617h37m5.743778808s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873688093 (617h37m5.744771128s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:05.873947867 (617h37m5.745030912s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Uploading piece... Dec 31 09:06:07.557273878 (617h37m7.428356913s) [upload=541230]: Next returned nil segment Dec 31 09:06:07.557296371 (617h37m7.428379416s) [upload=541230]: Waiting for error group managing segments... Dec 31 09:06:07.869989232 (617h37m7.741072267s) [noise=true , address=145.79.227.36:4519 , piece_id=K4MKJ3TFQE33LJ6VMTUJJTHR725C4CUX4KW3PCD73XKPTRCKMNAA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.877879937 (617h37m7.748962972s) [noise=true , address=145.79.227.47:4266 , piece_id=PPOTI2TRBUR3Y2VN23BEEHE5XHLXUOZLJZ6KWDEQAB3RTKNSNZ5A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.887580157 (617h37m7.758663203s) [noise=true , address=145.79.227.41:4086 , piece_id=RMA25UH4U6DVQLJWQLSVUVEJXO6NCU43PTIXMZTUAQAURWER47GA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894258915 (617h37m7.76534196s) [noise=true , address=145.79.227.38:4978 , piece_id=C343XQRJMHQ3O2F3UN7H26HYJ3ZA7FSE2REO6KJD734PD5YFPCAQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894331724 (617h37m7.765414759s) [noise=true , address=145.79.227.5:4222 , piece_id=MWRH3O4FG5ICW7JONZ4FYMKKIIXSYJZ5WPAWDH5R3OPRXNLDCCVA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.894725193 (617h37m7.765808238s) [noise=true , address=145.79.227.39:4014 , piece_id=FXY36EU2IXZXBX2SPU3GP6N3O25EFKV6PDJTYSIIIW77G6UAEKFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.933008700 (617h37m7.804091735s) [noise=true , address=145.79.227.16:4911 , piece_id=4VJTSRLYNOFOSVS3QOUROMUTEUMFW5DJ6ORRUSFSIIXORITHIWEQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:07.945354127 (617h37m7.816437172s) [noise=true , address=145.79.227.32:4834 , piece_id=ALGY4CXOJUYNK6BTWSW4O7JPC3RSUOF62M2DEWSXYUD5YX6ND7HA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.037141884 (617h37m7.908224929s) [noise=true , address=77.90.31.19:4842 , piece_id=LTE74CWDYFDNZIMZT4T5X4SSMSG27HYTV6ECSXM3E4MDLTY3R5CQ, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.105167495 (617h37m7.97625054s) [noise=true , address=77.90.31.11:4555 , piece_id=DUR2NOB43B572QCLAN5KLD53SNDX4XCSQJ5UJMYXHFGV7D3Y72RA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.156289979 (617h37m8.027373024s) [noise=true , address=77.90.31.13:4627 , piece_id=QADJI54EMPB54ZJXBWACVAR2APSUU5IGGGHVTG4K6AGS2YJUOHKA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.174675076 (617h37m8.045758111s) [noise=true , address=77.90.31.18:4805 , piece_id=5KOIVSWCCKOZCNXE5XLPHXBKXKATG3CWX52UM7YVYR4F5ONMH67A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188351619 (617h37m8.059434664s) [noise=true , address=45.91.116.6:4975 , piece_id=BQXVNM4A6MPZ2KAOLW6CNO7WER43K6YNAJ2LV23HLDOHFE6N7NNA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: <nil> Dec 31 09:06:08.188388489 (617h37m8.059471534s) [seg_pos={0 0}, upload=541230]: Segment reached optimal threshold of 13 pieces. Dec 31 09:06:08.188621363 (617h37m8.059704398s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188628386 (617h37m8.059711431s) [noise=true , address=45.91.116.10:4083 , piece_id=7HCHI74YRPT7Y7GM554KS7TZX3262DL5I2GYNOLKBGV7YRPY6NFA, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188762321 (617h37m8.059845356s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188778482 (617h37m8.059861517s) [noise=true , address=77.90.31.12:4589 , piece_id=72MDVLEZ2QHCEK22ICTXVK4LJOIYZPX3NALVTVRXZE4VTXF4AM6A, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188804071 (617h37m8.059887106s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 31 09:06:08.188811945 (617h37m8.05989498s) [noise=true , address=45.91.116.12:4228 , piece_id=WKYD7OZDWWMFNZHWBGMBKR4YJJ7H5X6MB343JHVQOUU6EGW6RE5Q, seg_pos={0 0}, upload=541230]: Long tail cancellation due to optimal threshold Dec 31 09:06:08.188858845 (617h37m8.05994188s) [seg_pos={0 0}, upload=541230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 09:06:08.189175598 (617h37m8.060258633s) [upload=541230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:06:08.189182111 (617h37m8.060265146s) [upload=541230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:06:08.483598230 (617h37m8.354681275s) [upload=541230]: Done uploading segments... Dec 31 09:06:08.483622396 (617h37m8.354705431s) [upload=541230]: Upload finished. err: <nil> Duration: 16.163961565s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...............X...........X....X....XXX.XX.............XX..] duration: 3.498766235s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:10:46 UTC) orbiter warning: context deadline exceeded Duration: 3.498792966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [................X...........X....X....XXX.XX.............XX.] duration: 3.406575502s trace: https://jaeger.dev.storj.io/trace/b3fe148e36f5660 FAIL (2025-12-31 09:10:31 UTC) orbiter warning: context deadline exceeded Duration: 3.406602754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 28.020504599s trace: https://jaeger.dev.storj.io/trace/26257de5bcb7c920 FAIL (2025-12-31 09:10:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== , upload=540795]: Uploading piece... Dec 31 08:35:48.315166374 (617h6m48.186249409s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.315823664 (617h6m48.186906699s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316244806 (617h6m48.187327841s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316251710 (617h6m48.187334755s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.316290584 (617h6m48.187373619s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317050111 (617h6m48.188133146s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.317140853 (617h6m48.188223888s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318123574 (617h6m48.189206609s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.318885204 (617h6m48.189968249s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.319835805 (617h6m48.19091885s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.320309847 (617h6m48.191392882s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321148273 (617h6m48.192231308s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.321502088 (617h6m48.192585133s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322658430 (617h6m48.193741495s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322775803 (617h6m48.193858848s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.322882035 (617h6m48.19396507s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:48.323403308 (617h6m48.194486343s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Uploading piece... Dec 31 08:35:49.768189393 (617h6m49.639272428s) [upload=540795]: Next returned nil segment Dec 31 08:35:49.768201797 (617h6m49.639284832s) [upload=540795]: Waiting for error group managing segments... Dec 31 08:35:50.197657834 (617h6m50.068740869s) [noise=true , address=145.79.227.16:4941 , piece_id=KDM5IUEVU5ZS5R3R63HXOOQCOBPLP7WQMWLXQFA6JQ2GQ2N6ESSQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.209847924 (617h6m50.08093096s) [noise=true , address=145.79.227.43:4163 , piece_id=J7VVPRY4KZ5GAMDUKVU5CTP4RB4X5EA7XIAO2ZZBFMFMT2UPSDNA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.217381621 (617h6m50.088464666s) [noise=true , address=145.79.227.5:4226 , piece_id=FKQWQTB3VGJJUX2SQONKZS42AK7VGT5MILYHYSPEHGUBOGIXBGPQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.270225653 (617h6m50.141308698s) [noise=true , address=145.79.227.35:4490 , piece_id=MZIAYRDR7QCA4PANQGBPSND34BMVCNKDSFBH552GXRPIJHXCZEBA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.411342630 (617h6m50.282425665s) [noise=true , address=77.90.31.10:4521 , piece_id=P3AYUPOJ5NJIXZ2MFX2YLBIRLZU6G754ILMX7RGLD6UZ2KWHFZYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473471693 (617h6m50.344554738s) [noise=true , address=77.90.31.12:4620 , piece_id=CA24NMLIPJZ6FOFK57VP5TZMKFCQZA75HR426RC3EZIIVUAWXWFQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.473607101 (617h6m50.344690136s) [noise=true , address=45.91.116.5:4305 , piece_id=76Z4ED4KZOVLYA5HWBNAXI6HOIS6NVV5O5267OLPP35UQO7GYLCA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.479010038 (617h6m50.350093074s) [noise=true , address=77.90.31.21:4891 , piece_id=VSJHZWJFAGHIT36PZCTG3JO5F7SFY43RVG4TL2PTFV6I3MKCKY3A, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.482534590 (617h6m50.353617625s) [noise=true , address=77.90.31.15:4713 , piece_id=ASPILFEZZN4EOKBGYOLWETULZSDUCNNAVEPH37FXPPOPKUT4YCUQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:50.485954982 (617h6m50.357038017s) [noise=true , address=77.90.31.16:4748 , piece_id=XN4OAU4WEEM2B45G5AT5C2GHQ3Y4CKYXUD52VEXXNWP53V7EOXYA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.058742323 (617h6m50.929825358s) [noise=true , address=145.79.227.44:4219 , piece_id=2QBWX6QQCJ42RA66WOINBPAO34I4LHYVCW2NSB2MIIAT2QT3NOWA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.071853740 (617h6m50.942936775s) [noise=true , address=145.79.227.30:4769 , piece_id=4ZYRDTCH2VNGPZ57AWM6STXEA4G676BZJJJ2272JBXJZJ7BXUOQA, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.177856875 (617h6m51.04893991s) [noise=true , address=145.79.227.33:4873 , piece_id=KWLS4DUDNAEADPUA3CTATKK4MEQSUPEIPYJS3SSOGVPDOJPJHDIQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179660939 (617h6m51.050743984s) [noise=true , address=145.79.227.29:4957 , piece_id=PKUBQTLYRGZHJWQBTA5K22P32Q3LBXFVGPRMEXXWVRX2HJERFI6Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: <nil> Dec 31 08:35:51.179686989 (617h6m51.050770024s) [seg_pos={0 0}, upload=540795]: Segment reached optimal threshold of 14 pieces. Dec 31 08:35:51.179944619 (617h6m51.051027684s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.179956683 (617h6m51.051039728s) [noise=true , address=145.79.227.49:4345 , piece_id=Y7BV6CVRWUYQQSM6X3LJ2PTMELRWZXMY5ZKTPULXHG7JOKYSUB2Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180323110 (617h6m51.051406155s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180346325 (617h6m51.05142937s) [noise=true , address=145.79.227.34:4908 , piece_id=QKCE7MJA25VM2TYEJALOJXJUJJUITJQN4ARDCQCCDW4UHI2CQBXQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180485580 (617h6m51.051568615s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180495899 (617h6m51.051578934s) [noise=true , address=45.91.116.14:4194 , piece_id=A35OICEK3K3BGPMDENNJKQAQUQ2UKBVJLWG5UR4VSFSJ2GIHF2NQ, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180529313 (617h6m51.051612358s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 31 08:35:51.180536927 (617h6m51.051619972s) [noise=true , address=145.79.227.40:4076 , piece_id=JCGS42W3KNQKMNOTIR7D4SQ4RPHAFEAQ4VAELMCCTVTDK4JUC46Q, seg_pos={0 0}, upload=540795]: Long tail cancellation due to optimal threshold Dec 31 08:35:51.180560653 (617h6m51.051643698s) [seg_pos={0 0}, upload=540795]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 08:35:51.180641827 (617h6m51.051724872s) [upload=540795]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 08:35:51.180647598 (617h6m51.051730643s) [upload=540795]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 08:35:51.479648850 (617h6m51.350731896s) [upload=540795]: Done uploading segments... Dec 31 08:35:51.479871193 (617h6m51.350954228s) [upload=540795]: Upload finished. err: <nil> Duration: 28.020532894s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)