Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X...........X.....X...X.....XX.X.....X......XX..............] duration: 3.48215038s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 06:04:09 UTC) orbiter warning: context deadline exceeded Duration: 3.482178916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X....................................X.X...................] duration: 15.433709222s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 06:03:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 15.433753225s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X...XX...X.XX...........XXXX....X.X.X..X...........X.X..X..] duration: 6.061922814s trace: https://jaeger.dev.storj.io/trace/21df6c334259b2b9 FAIL (2025-12-29 06:03:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: HVOAPYR2I75PI4DDY72GYVUG3CZS6X3ZRQ35HG5PBHNFXOD5AWQQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7VINOFFJKK33AXVI7XYLYH2K6DVRJ3MKPZ7FZTRKQDAH2BCPQGTA) context deadline exceeded ======LOGS====== segments... Dec 29 06:01:32.143008453 (566h32m32.014091488s) [upload=496717]: Got next segment. Inline: false Dec 29 06:01:32.143085740 (566h32m32.014168786s) [upload=496717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:01:32.143092744 (566h32m32.014175789s) [upload=496717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:01:32.404336082 (566h32m32.275419127s) [seg_pos={0 0}, upload=496717]: Begin upload segment... Dec 29 06:01:32.404399502 (566h32m32.275482537s) [seg_pos={0 0}, upload=496717]: Done begin upload segment. Dec 29 06:01:32.404464636 (566h32m32.275547672s) [noise=true , address=145.79.227.37:4566 , piece_id=QTFJOGKBBVY2UVGFCL4KXARJNGFEAIYEB53UOQDFCZEACAOSXOSA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.404473755 (566h32m32.27555679s) [noise=true , address=77.90.31.11:4553 , piece_id=HVOAPYR2I75PI4DDY72GYVUG3CZS6X3ZRQ35HG5PBHNFXOD5AWQQ, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.405129583 (566h32m32.276212618s) [noise=true , address=145.79.227.46:4602 , piece_id=FDG72FRLX7H7SSA52CF4DFU6EKF45JRZ7QFLKH53TRAY2BFOFRCQ, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.405563499 (566h32m32.276646534s) [noise=true , address=145.79.227.41:4087 , piece_id=2HRUEAXD6FLSFVE4E56V54HLV4Q4DAIUQ2CDIE5RFWXNOBQQY43A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.405984641 (566h32m32.277067686s) [noise=true , address=145.79.227.16:4911 , piece_id=UIYXD2KI6MZ6UQFWBL7ABMQDWBWF5KBVNXQC24RHXVNO3OUTGLDA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.406480736 (566h32m32.277563771s) [noise=true , address=45.91.116.12:4228 , piece_id=MCDQD76SOQ5NY6QJJQDNVI6CKTWJDJUDUJZA6KDQVZNJKDTC3M2A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.406936924 (566h32m32.278019959s) [noise=true , address=145.79.227.38:4978 , piece_id=4E4CLHBFD4LEF2UMBHSDWGCYQEPRWR7QWSIN4S5U7XEBULWLYJMA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.407420575 (566h32m32.27850361s) [noise=true , address=45.91.116.5:4299 , piece_id=7VINOFFJKK33AXVI7XYLYH2K6DVRJ3MKPZ7FZTRKQDAH2BCPQGTA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.408703037 (566h32m32.279786072s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSNT7YONWR6EHTPM4Z3AC5VF6F32FU5HI67A664IPPR2CA45QI4A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.408877469 (566h32m32.279960515s) [noise=true , address=45.91.116.11:4117 , piece_id=W6Y7AAHZG52VJS5VSZYE5DGIEYA4AXLCUFGA5MRDVPGVP4O5MD2A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.412871695 (566h32m32.28395473s) [noise=true , address=45.91.116.6:4974 , piece_id=VFAS73EC3TVH7VXIBCVNKMJLRYDA7PAD4SZUBZXSSLGNDHLLK3IA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.415542751 (566h32m32.286625786s) [noise=true , address=145.79.227.14:4840 , piece_id=SUVLLZJCQEFSYRPWSEAECNBJW4MKJUIW3STASIITTO4C3NOPBVWA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.418216572 (566h32m32.289299608s) [noise=true , address=145.79.227.24:4582 , piece_id=VL632JZGQMXBJ7QWD2GG62ND5WXBLS7O5WLLWALJDH2XQLDKJZNA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.421601518 (566h32m32.292684553s) [noise=true , address=45.91.116.14:4190 , piece_id=JZ4TU2RHRAYSPK6HC7WYNAKSTRZJTUZGFF2IR32WERVWLQIRQ7AA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.423300523 (566h32m32.294383558s) [noise=true , address=145.79.227.51:4410 , piece_id=77ACJDD72WGN4XSCZS7S5ROGXBF265N2ASND22NVG6A43KKSNLKQ, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.425774724 (566h32m32.296857769s) [noise=true , address=145.79.227.2:4113 , piece_id=VATF6BNCT7M5AX2RIEN2PDYSLCDNWKVYDI66H3R5SZHHNVAAWT4Q, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:34.113777296 (566h32m33.984860341s) [upload=496717]: Next returned nil segment Dec 29 06:01:34.113789089 (566h32m33.984872124s) [upload=496717]: Waiting for error group managing segments... Dec 29 06:01:34.396125641 (566h32m34.267208686s) [noise=true , address=145.79.227.16:4911 , piece_id=UIYXD2KI6MZ6UQFWBL7ABMQDWBWF5KBVNXQC24RHXVNO3OUTGLDA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.396618661 (566h32m34.267701706s) [noise=true , address=145.79.227.41:4087 , piece_id=2HRUEAXD6FLSFVE4E56V54HLV4Q4DAIUQ2CDIE5RFWXNOBQQY43A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.397943573 (566h32m34.269026618s) [noise=true , address=145.79.227.46:4602 , piece_id=FDG72FRLX7H7SSA52CF4DFU6EKF45JRZ7QFLKH53TRAY2BFOFRCQ, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.402184097 (566h32m34.273267132s) [noise=true , address=145.79.227.51:4410 , piece_id=77ACJDD72WGN4XSCZS7S5ROGXBF265N2ASND22NVG6A43KKSNLKQ, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.403666731 (566h32m34.274749776s) [noise=true , address=145.79.227.38:4978 , piece_id=4E4CLHBFD4LEF2UMBHSDWGCYQEPRWR7QWSIN4S5U7XEBULWLYJMA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.405825722 (566h32m34.276908767s) [noise=true , address=145.79.227.2:4113 , piece_id=VATF6BNCT7M5AX2RIEN2PDYSLCDNWKVYDI66H3R5SZHHNVAAWT4Q, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.410557672 (566h32m34.281640707s) [noise=true , address=145.79.227.37:4566 , piece_id=QTFJOGKBBVY2UVGFCL4KXARJNGFEAIYEB53UOQDFCZEACAOSXOSA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.416568348 (566h32m34.287651383s) [noise=true , address=145.79.227.14:4840 , piece_id=SUVLLZJCQEFSYRPWSEAECNBJW4MKJUIW3STASIITTO4C3NOPBVWA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.417143863 (566h32m34.288226899s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSNT7YONWR6EHTPM4Z3AC5VF6F32FU5HI67A664IPPR2CA45QI4A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.428831128 (566h32m34.299914163s) [noise=true , address=145.79.227.24:4582 , piece_id=VL632JZGQMXBJ7QWD2GG62ND5WXBLS7O5WLLWALJDH2XQLDKJZNA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.617037272 (566h32m34.488120317s) [noise=true , address=77.90.31.11:4553 , piece_id=HVOAPYR2I75PI4DDY72GYVUG3CZS6X3ZRQ35HG5PBHNFXOD5AWQQ, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.629086065 (566h32m34.50016911s) [noise=true , address=45.91.116.5:4299 , piece_id=7VINOFFJKK33AXVI7XYLYH2K6DVRJ3MKPZ7FZTRKQDAH2BCPQGTA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.755682484 (566h32m34.626765519s) [noise=true , address=45.91.116.12:4228 , piece_id=MCDQD76SOQ5NY6QJJQDNVI6CKTWJDJUDUJZA6KDQVZNJKDTC3M2A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.755702642 (566h32m34.626785677s) [seg_pos={0 0}, upload=496717]: Segment reached optimal threshold of 13 pieces. Dec 29 06:01:34.755897333 (566h32m34.626980368s) [noise=true , address=45.91.116.14:4190 , piece_id=JZ4TU2RHRAYSPK6HC7WYNAKSTRZJTUZGFF2IR32WERVWLQIRQ7AA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 06:01:34.755906901 (566h32m34.626989936s) [noise=true , address=45.91.116.14:4190 , piece_id=JZ4TU2RHRAYSPK6HC7WYNAKSTRZJTUZGFF2IR32WERVWLQIRQ7AA, seg_pos={0 0}, upload=496717]: Long tail cancellation due to optimal threshold Dec 29 06:01:34.756043131 (566h32m34.627126166s) [noise=true , address=45.91.116.11:4117 , piece_id=W6Y7AAHZG52VJS5VSZYE5DGIEYA4AXLCUFGA5MRDVPGVP4O5MD2A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:45120->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:45120->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->45.91.116.11:4117: use of closed network connection Dec 29 06:01:34.756049042 (566h32m34.627132087s) [noise=true , address=45.91.116.11:4117 , piece_id=W6Y7AAHZG52VJS5VSZYE5DGIEYA4AXLCUFGA5MRDVPGVP4O5MD2A, seg_pos={0 0}, upload=496717]: Long tail cancellation due to optimal threshold Dec 29 06:01:34.756202485 (566h32m34.62728552s) [noise=true , address=45.91.116.6:4974 , piece_id=VFAS73EC3TVH7VXIBCVNKMJLRYDA7PAD4SZUBZXSSLGNDHLLK3IA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:46260->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:46260->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46260->45.91.116.6:4974: use of closed network connection Dec 29 06:01:34.756220318 (566h32m34.627303363s) [noise=true , address=45.91.116.6:4974 , piece_id=VFAS73EC3TVH7VXIBCVNKMJLRYDA7PAD4SZUBZXSSLGNDHLLK3IA, seg_pos={0 0}, upload=496717]: Long tail cancellation due to optimal threshold Dec 29 06:01:34.756264592 (566h32m34.627347637s) [seg_pos={0 0}, upload=496717]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:01:34.756374010 (566h32m34.627457055s) [upload=496717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:01:34.756381425 (566h32m34.62746447s) [upload=496717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:01:35.043768326 (566h32m34.914851371s) [upload=496717]: Done uploading segments... Dec 29 06:01:35.043794105 (566h32m34.91487715s) [upload=496717]: Upload finished. err: <nil> Duration: 6.061951628s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X...........X.....X...X.....XX.X.....X......XX.............] duration: 3.442156426s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 06:02:57 UTC) orbiter warning: context deadline exceeded Duration: 3.442181163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X...XX...X.XX...........XXXX....X.X.X..X...........X.X..X.] duration: 5.984010116s trace: https://jaeger.dev.storj.io/trace/21df6c334259b2b9 FAIL (2025-12-29 06:02:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: HVOAPYR2I75PI4DDY72GYVUG3CZS6X3ZRQ35HG5PBHNFXOD5AWQQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7VINOFFJKK33AXVI7XYLYH2K6DVRJ3MKPZ7FZTRKQDAH2BCPQGTA) context deadline exceeded ======LOGS====== segments... Dec 29 06:01:32.143008453 (566h32m32.014091488s) [upload=496717]: Got next segment. Inline: false Dec 29 06:01:32.143085740 (566h32m32.014168786s) [upload=496717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:01:32.143092744 (566h32m32.014175789s) [upload=496717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:01:32.404336082 (566h32m32.275419127s) [seg_pos={0 0}, upload=496717]: Begin upload segment... Dec 29 06:01:32.404399502 (566h32m32.275482537s) [seg_pos={0 0}, upload=496717]: Done begin upload segment. Dec 29 06:01:32.404464636 (566h32m32.275547672s) [noise=true , address=145.79.227.37:4566 , piece_id=QTFJOGKBBVY2UVGFCL4KXARJNGFEAIYEB53UOQDFCZEACAOSXOSA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.404473755 (566h32m32.27555679s) [noise=true , address=77.90.31.11:4553 , piece_id=HVOAPYR2I75PI4DDY72GYVUG3CZS6X3ZRQ35HG5PBHNFXOD5AWQQ, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.405129583 (566h32m32.276212618s) [noise=true , address=145.79.227.46:4602 , piece_id=FDG72FRLX7H7SSA52CF4DFU6EKF45JRZ7QFLKH53TRAY2BFOFRCQ, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.405563499 (566h32m32.276646534s) [noise=true , address=145.79.227.41:4087 , piece_id=2HRUEAXD6FLSFVE4E56V54HLV4Q4DAIUQ2CDIE5RFWXNOBQQY43A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.405984641 (566h32m32.277067686s) [noise=true , address=145.79.227.16:4911 , piece_id=UIYXD2KI6MZ6UQFWBL7ABMQDWBWF5KBVNXQC24RHXVNO3OUTGLDA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.406480736 (566h32m32.277563771s) [noise=true , address=45.91.116.12:4228 , piece_id=MCDQD76SOQ5NY6QJJQDNVI6CKTWJDJUDUJZA6KDQVZNJKDTC3M2A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.406936924 (566h32m32.278019959s) [noise=true , address=145.79.227.38:4978 , piece_id=4E4CLHBFD4LEF2UMBHSDWGCYQEPRWR7QWSIN4S5U7XEBULWLYJMA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.407420575 (566h32m32.27850361s) [noise=true , address=45.91.116.5:4299 , piece_id=7VINOFFJKK33AXVI7XYLYH2K6DVRJ3MKPZ7FZTRKQDAH2BCPQGTA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.408703037 (566h32m32.279786072s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSNT7YONWR6EHTPM4Z3AC5VF6F32FU5HI67A664IPPR2CA45QI4A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.408877469 (566h32m32.279960515s) [noise=true , address=45.91.116.11:4117 , piece_id=W6Y7AAHZG52VJS5VSZYE5DGIEYA4AXLCUFGA5MRDVPGVP4O5MD2A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.412871695 (566h32m32.28395473s) [noise=true , address=45.91.116.6:4974 , piece_id=VFAS73EC3TVH7VXIBCVNKMJLRYDA7PAD4SZUBZXSSLGNDHLLK3IA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.415542751 (566h32m32.286625786s) [noise=true , address=145.79.227.14:4840 , piece_id=SUVLLZJCQEFSYRPWSEAECNBJW4MKJUIW3STASIITTO4C3NOPBVWA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.418216572 (566h32m32.289299608s) [noise=true , address=145.79.227.24:4582 , piece_id=VL632JZGQMXBJ7QWD2GG62ND5WXBLS7O5WLLWALJDH2XQLDKJZNA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.421601518 (566h32m32.292684553s) [noise=true , address=45.91.116.14:4190 , piece_id=JZ4TU2RHRAYSPK6HC7WYNAKSTRZJTUZGFF2IR32WERVWLQIRQ7AA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.423300523 (566h32m32.294383558s) [noise=true , address=145.79.227.51:4410 , piece_id=77ACJDD72WGN4XSCZS7S5ROGXBF265N2ASND22NVG6A43KKSNLKQ, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.425774724 (566h32m32.296857769s) [noise=true , address=145.79.227.2:4113 , piece_id=VATF6BNCT7M5AX2RIEN2PDYSLCDNWKVYDI66H3R5SZHHNVAAWT4Q, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:34.113777296 (566h32m33.984860341s) [upload=496717]: Next returned nil segment Dec 29 06:01:34.113789089 (566h32m33.984872124s) [upload=496717]: Waiting for error group managing segments... Dec 29 06:01:34.396125641 (566h32m34.267208686s) [noise=true , address=145.79.227.16:4911 , piece_id=UIYXD2KI6MZ6UQFWBL7ABMQDWBWF5KBVNXQC24RHXVNO3OUTGLDA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.396618661 (566h32m34.267701706s) [noise=true , address=145.79.227.41:4087 , piece_id=2HRUEAXD6FLSFVE4E56V54HLV4Q4DAIUQ2CDIE5RFWXNOBQQY43A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.397943573 (566h32m34.269026618s) [noise=true , address=145.79.227.46:4602 , piece_id=FDG72FRLX7H7SSA52CF4DFU6EKF45JRZ7QFLKH53TRAY2BFOFRCQ, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.402184097 (566h32m34.273267132s) [noise=true , address=145.79.227.51:4410 , piece_id=77ACJDD72WGN4XSCZS7S5ROGXBF265N2ASND22NVG6A43KKSNLKQ, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.403666731 (566h32m34.274749776s) [noise=true , address=145.79.227.38:4978 , piece_id=4E4CLHBFD4LEF2UMBHSDWGCYQEPRWR7QWSIN4S5U7XEBULWLYJMA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.405825722 (566h32m34.276908767s) [noise=true , address=145.79.227.2:4113 , piece_id=VATF6BNCT7M5AX2RIEN2PDYSLCDNWKVYDI66H3R5SZHHNVAAWT4Q, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.410557672 (566h32m34.281640707s) [noise=true , address=145.79.227.37:4566 , piece_id=QTFJOGKBBVY2UVGFCL4KXARJNGFEAIYEB53UOQDFCZEACAOSXOSA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.416568348 (566h32m34.287651383s) [noise=true , address=145.79.227.14:4840 , piece_id=SUVLLZJCQEFSYRPWSEAECNBJW4MKJUIW3STASIITTO4C3NOPBVWA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.417143863 (566h32m34.288226899s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSNT7YONWR6EHTPM4Z3AC5VF6F32FU5HI67A664IPPR2CA45QI4A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.428831128 (566h32m34.299914163s) [noise=true , address=145.79.227.24:4582 , piece_id=VL632JZGQMXBJ7QWD2GG62ND5WXBLS7O5WLLWALJDH2XQLDKJZNA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.617037272 (566h32m34.488120317s) [noise=true , address=77.90.31.11:4553 , piece_id=HVOAPYR2I75PI4DDY72GYVUG3CZS6X3ZRQ35HG5PBHNFXOD5AWQQ, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.629086065 (566h32m34.50016911s) [noise=true , address=45.91.116.5:4299 , piece_id=7VINOFFJKK33AXVI7XYLYH2K6DVRJ3MKPZ7FZTRKQDAH2BCPQGTA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.755682484 (566h32m34.626765519s) [noise=true , address=45.91.116.12:4228 , piece_id=MCDQD76SOQ5NY6QJJQDNVI6CKTWJDJUDUJZA6KDQVZNJKDTC3M2A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.755702642 (566h32m34.626785677s) [seg_pos={0 0}, upload=496717]: Segment reached optimal threshold of 13 pieces. Dec 29 06:01:34.755897333 (566h32m34.626980368s) [noise=true , address=45.91.116.14:4190 , piece_id=JZ4TU2RHRAYSPK6HC7WYNAKSTRZJTUZGFF2IR32WERVWLQIRQ7AA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 06:01:34.755906901 (566h32m34.626989936s) [noise=true , address=45.91.116.14:4190 , piece_id=JZ4TU2RHRAYSPK6HC7WYNAKSTRZJTUZGFF2IR32WERVWLQIRQ7AA, seg_pos={0 0}, upload=496717]: Long tail cancellation due to optimal threshold Dec 29 06:01:34.756043131 (566h32m34.627126166s) [noise=true , address=45.91.116.11:4117 , piece_id=W6Y7AAHZG52VJS5VSZYE5DGIEYA4AXLCUFGA5MRDVPGVP4O5MD2A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:45120->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:45120->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->45.91.116.11:4117: use of closed network connection Dec 29 06:01:34.756049042 (566h32m34.627132087s) [noise=true , address=45.91.116.11:4117 , piece_id=W6Y7AAHZG52VJS5VSZYE5DGIEYA4AXLCUFGA5MRDVPGVP4O5MD2A, seg_pos={0 0}, upload=496717]: Long tail cancellation due to optimal threshold Dec 29 06:01:34.756202485 (566h32m34.62728552s) [noise=true , address=45.91.116.6:4974 , piece_id=VFAS73EC3TVH7VXIBCVNKMJLRYDA7PAD4SZUBZXSSLGNDHLLK3IA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:46260->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:46260->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46260->45.91.116.6:4974: use of closed network connection Dec 29 06:01:34.756220318 (566h32m34.627303363s) [noise=true , address=45.91.116.6:4974 , piece_id=VFAS73EC3TVH7VXIBCVNKMJLRYDA7PAD4SZUBZXSSLGNDHLLK3IA, seg_pos={0 0}, upload=496717]: Long tail cancellation due to optimal threshold Dec 29 06:01:34.756264592 (566h32m34.627347637s) [seg_pos={0 0}, upload=496717]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:01:34.756374010 (566h32m34.627457055s) [upload=496717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:01:34.756381425 (566h32m34.62746447s) [upload=496717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:01:35.043768326 (566h32m34.914851371s) [upload=496717]: Done uploading segments... Dec 29 06:01:35.043794105 (566h32m34.91487715s) [upload=496717]: Upload finished. err: <nil> Duration: 5.984026657s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X....................................X.X..................] duration: 30.415190772s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 06:02:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 30.415221812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.X...XX...X.XX...........XXXX....X.X.X..X...........X.X..X] duration: 30.003662453s trace: https://jaeger.dev.storj.io/trace/21df6c334259b2b9 FAIL (2025-12-29 06:02:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: HVOAPYR2I75PI4DDY72GYVUG3CZS6X3ZRQ35HG5PBHNFXOD5AWQQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7VINOFFJKK33AXVI7XYLYH2K6DVRJ3MKPZ7FZTRKQDAH2BCPQGTA) context deadline exceeded ======LOGS====== segments... Dec 29 06:01:32.143008453 (566h32m32.014091488s) [upload=496717]: Got next segment. Inline: false Dec 29 06:01:32.143085740 (566h32m32.014168786s) [upload=496717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 06:01:32.143092744 (566h32m32.014175789s) [upload=496717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 06:01:32.404336082 (566h32m32.275419127s) [seg_pos={0 0}, upload=496717]: Begin upload segment... Dec 29 06:01:32.404399502 (566h32m32.275482537s) [seg_pos={0 0}, upload=496717]: Done begin upload segment. Dec 29 06:01:32.404464636 (566h32m32.275547672s) [noise=true , address=145.79.227.37:4566 , piece_id=QTFJOGKBBVY2UVGFCL4KXARJNGFEAIYEB53UOQDFCZEACAOSXOSA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.404473755 (566h32m32.27555679s) [noise=true , address=77.90.31.11:4553 , piece_id=HVOAPYR2I75PI4DDY72GYVUG3CZS6X3ZRQ35HG5PBHNFXOD5AWQQ, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.405129583 (566h32m32.276212618s) [noise=true , address=145.79.227.46:4602 , piece_id=FDG72FRLX7H7SSA52CF4DFU6EKF45JRZ7QFLKH53TRAY2BFOFRCQ, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.405563499 (566h32m32.276646534s) [noise=true , address=145.79.227.41:4087 , piece_id=2HRUEAXD6FLSFVE4E56V54HLV4Q4DAIUQ2CDIE5RFWXNOBQQY43A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.405984641 (566h32m32.277067686s) [noise=true , address=145.79.227.16:4911 , piece_id=UIYXD2KI6MZ6UQFWBL7ABMQDWBWF5KBVNXQC24RHXVNO3OUTGLDA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.406480736 (566h32m32.277563771s) [noise=true , address=45.91.116.12:4228 , piece_id=MCDQD76SOQ5NY6QJJQDNVI6CKTWJDJUDUJZA6KDQVZNJKDTC3M2A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.406936924 (566h32m32.278019959s) [noise=true , address=145.79.227.38:4978 , piece_id=4E4CLHBFD4LEF2UMBHSDWGCYQEPRWR7QWSIN4S5U7XEBULWLYJMA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.407420575 (566h32m32.27850361s) [noise=true , address=45.91.116.5:4299 , piece_id=7VINOFFJKK33AXVI7XYLYH2K6DVRJ3MKPZ7FZTRKQDAH2BCPQGTA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.408703037 (566h32m32.279786072s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSNT7YONWR6EHTPM4Z3AC5VF6F32FU5HI67A664IPPR2CA45QI4A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.408877469 (566h32m32.279960515s) [noise=true , address=45.91.116.11:4117 , piece_id=W6Y7AAHZG52VJS5VSZYE5DGIEYA4AXLCUFGA5MRDVPGVP4O5MD2A, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.412871695 (566h32m32.28395473s) [noise=true , address=45.91.116.6:4974 , piece_id=VFAS73EC3TVH7VXIBCVNKMJLRYDA7PAD4SZUBZXSSLGNDHLLK3IA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.415542751 (566h32m32.286625786s) [noise=true , address=145.79.227.14:4840 , piece_id=SUVLLZJCQEFSYRPWSEAECNBJW4MKJUIW3STASIITTO4C3NOPBVWA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.418216572 (566h32m32.289299608s) [noise=true , address=145.79.227.24:4582 , piece_id=VL632JZGQMXBJ7QWD2GG62ND5WXBLS7O5WLLWALJDH2XQLDKJZNA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.421601518 (566h32m32.292684553s) [noise=true , address=45.91.116.14:4190 , piece_id=JZ4TU2RHRAYSPK6HC7WYNAKSTRZJTUZGFF2IR32WERVWLQIRQ7AA, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.423300523 (566h32m32.294383558s) [noise=true , address=145.79.227.51:4410 , piece_id=77ACJDD72WGN4XSCZS7S5ROGXBF265N2ASND22NVG6A43KKSNLKQ, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:32.425774724 (566h32m32.296857769s) [noise=true , address=145.79.227.2:4113 , piece_id=VATF6BNCT7M5AX2RIEN2PDYSLCDNWKVYDI66H3R5SZHHNVAAWT4Q, seg_pos={0 0}, upload=496717]: Uploading piece... Dec 29 06:01:34.113777296 (566h32m33.984860341s) [upload=496717]: Next returned nil segment Dec 29 06:01:34.113789089 (566h32m33.984872124s) [upload=496717]: Waiting for error group managing segments... Dec 29 06:01:34.396125641 (566h32m34.267208686s) [noise=true , address=145.79.227.16:4911 , piece_id=UIYXD2KI6MZ6UQFWBL7ABMQDWBWF5KBVNXQC24RHXVNO3OUTGLDA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.396618661 (566h32m34.267701706s) [noise=true , address=145.79.227.41:4087 , piece_id=2HRUEAXD6FLSFVE4E56V54HLV4Q4DAIUQ2CDIE5RFWXNOBQQY43A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.397943573 (566h32m34.269026618s) [noise=true , address=145.79.227.46:4602 , piece_id=FDG72FRLX7H7SSA52CF4DFU6EKF45JRZ7QFLKH53TRAY2BFOFRCQ, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.402184097 (566h32m34.273267132s) [noise=true , address=145.79.227.51:4410 , piece_id=77ACJDD72WGN4XSCZS7S5ROGXBF265N2ASND22NVG6A43KKSNLKQ, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.403666731 (566h32m34.274749776s) [noise=true , address=145.79.227.38:4978 , piece_id=4E4CLHBFD4LEF2UMBHSDWGCYQEPRWR7QWSIN4S5U7XEBULWLYJMA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.405825722 (566h32m34.276908767s) [noise=true , address=145.79.227.2:4113 , piece_id=VATF6BNCT7M5AX2RIEN2PDYSLCDNWKVYDI66H3R5SZHHNVAAWT4Q, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.410557672 (566h32m34.281640707s) [noise=true , address=145.79.227.37:4566 , piece_id=QTFJOGKBBVY2UVGFCL4KXARJNGFEAIYEB53UOQDFCZEACAOSXOSA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.416568348 (566h32m34.287651383s) [noise=true , address=145.79.227.14:4840 , piece_id=SUVLLZJCQEFSYRPWSEAECNBJW4MKJUIW3STASIITTO4C3NOPBVWA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.417143863 (566h32m34.288226899s) [noise=true , address=145.79.227.50:4374 , piece_id=ZSNT7YONWR6EHTPM4Z3AC5VF6F32FU5HI67A664IPPR2CA45QI4A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.428831128 (566h32m34.299914163s) [noise=true , address=145.79.227.24:4582 , piece_id=VL632JZGQMXBJ7QWD2GG62ND5WXBLS7O5WLLWALJDH2XQLDKJZNA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.617037272 (566h32m34.488120317s) [noise=true , address=77.90.31.11:4553 , piece_id=HVOAPYR2I75PI4DDY72GYVUG3CZS6X3ZRQ35HG5PBHNFXOD5AWQQ, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.629086065 (566h32m34.50016911s) [noise=true , address=45.91.116.5:4299 , piece_id=7VINOFFJKK33AXVI7XYLYH2K6DVRJ3MKPZ7FZTRKQDAH2BCPQGTA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.755682484 (566h32m34.626765519s) [noise=true , address=45.91.116.12:4228 , piece_id=MCDQD76SOQ5NY6QJJQDNVI6CKTWJDJUDUJZA6KDQVZNJKDTC3M2A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: <nil> Dec 29 06:01:34.755702642 (566h32m34.626785677s) [seg_pos={0 0}, upload=496717]: Segment reached optimal threshold of 13 pieces. Dec 29 06:01:34.755897333 (566h32m34.626980368s) [noise=true , address=45.91.116.14:4190 , piece_id=JZ4TU2RHRAYSPK6HC7WYNAKSTRZJTUZGFF2IR32WERVWLQIRQ7AA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 06:01:34.755906901 (566h32m34.626989936s) [noise=true , address=45.91.116.14:4190 , piece_id=JZ4TU2RHRAYSPK6HC7WYNAKSTRZJTUZGFF2IR32WERVWLQIRQ7AA, seg_pos={0 0}, upload=496717]: Long tail cancellation due to optimal threshold Dec 29 06:01:34.756043131 (566h32m34.627126166s) [noise=true , address=45.91.116.11:4117 , piece_id=W6Y7AAHZG52VJS5VSZYE5DGIEYA4AXLCUFGA5MRDVPGVP4O5MD2A, seg_pos={0 0}, upload=496717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:45120->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:45120->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45120->45.91.116.11:4117: use of closed network connection Dec 29 06:01:34.756049042 (566h32m34.627132087s) [noise=true , address=45.91.116.11:4117 , piece_id=W6Y7AAHZG52VJS5VSZYE5DGIEYA4AXLCUFGA5MRDVPGVP4O5MD2A, seg_pos={0 0}, upload=496717]: Long tail cancellation due to optimal threshold Dec 29 06:01:34.756202485 (566h32m34.62728552s) [noise=true , address=45.91.116.6:4974 , piece_id=VFAS73EC3TVH7VXIBCVNKMJLRYDA7PAD4SZUBZXSSLGNDHLLK3IA, seg_pos={0 0}, upload=496717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:46260->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:46260->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46260->45.91.116.6:4974: use of closed network connection Dec 29 06:01:34.756220318 (566h32m34.627303363s) [noise=true , address=45.91.116.6:4974 , piece_id=VFAS73EC3TVH7VXIBCVNKMJLRYDA7PAD4SZUBZXSSLGNDHLLK3IA, seg_pos={0 0}, upload=496717]: Long tail cancellation due to optimal threshold Dec 29 06:01:34.756264592 (566h32m34.627347637s) [seg_pos={0 0}, upload=496717]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 06:01:34.756374010 (566h32m34.627457055s) [upload=496717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 06:01:34.756381425 (566h32m34.62746447s) [upload=496717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 06:01:35.043768326 (566h32m34.914851371s) [upload=496717]: Done uploading segments... Dec 29 06:01:35.043794105 (566h32m34.91487715s) [upload=496717]: Upload finished. err: <nil> Duration: 30.003688601s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..X...........X.....X...X.....XX.X.....X......XX............] duration: 3.482256615s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 06:01:53 UTC) orbiter warning: context deadline exceeded Duration: 3.482283305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X....................................X.X.................] duration: 36.093907181s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 06:01:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 36.093941406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...X....................................X.X................] duration: 15.82096028s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 06:00:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 15.820998883s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...X...........X.....X...X.....XX.X.....X......XX...........] duration: 3.609176321s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 06:00:56 UTC) orbiter warning: context deadline exceeded Duration: 3.609204336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.X...XX...X.XX...........XXXX....X.X.X..X...........X.X..] duration: 6.04256745s trace: https://jaeger.dev.storj.io/trace/27e4aa47141ed4e2 FAIL (2025-12-29 06:00:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Dec 29 05:58:31.521946993 (566h29m31.393030028s) [upload=496676]: Got next segment. Inline: false Dec 29 05:58:31.522000294 (566h29m31.393083329s) [upload=496676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:58:31.522004933 (566h29m31.393087968s) [upload=496676]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:58:31.769206577 (566h29m31.640289622s) [seg_pos={0 0}, upload=496676]: Begin upload segment... Dec 29 05:58:31.769250261 (566h29m31.640333296s) [seg_pos={0 0}, upload=496676]: Done begin upload segment. Dec 29 05:58:31.769274075 (566h29m31.640357121s) [noise=true , address=145.79.227.47:4266 , piece_id=M3QWVIYBOAWZ7ODPFRRMFF77DY6CSRH6ACYKS6UHY5MK47I4OCHA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.769995219 (566h29m31.641078264s) [noise=true , address=77.90.31.19:4842 , piece_id=7V5WEYXV5ZKSUZF5Y7SWYWX7NY3K7QBECQLHEOQ5QMTNPPRAND4A, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.770470194 (566h29m31.641553239s) [noise=true , address=145.79.227.12:4765 , piece_id=2NV3ISJZMR2ICAYCFUGZFPRKJ2YOOC3LGFD6WMVNW6WB4ECZ4M3Q, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.770834607 (566h29m31.641917642s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZ5VETOVJGQ46FO6BWOCKRI7C6ZQPI6A4K75F57QDAD7XOELZOQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.771077490 (566h29m31.642160525s) [noise=true , address=77.90.31.16:4735 , piece_id=O7FYVZSBSF44FEVF6WUH4LMAGRXM7M4JXD3BKDBEIEC4JFACMQ5Q, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.771490156 (566h29m31.642573201s) [noise=true , address=145.79.227.46:4601 , piece_id=GGJTW5DEHWQ2XKDPWGDMBDM3U5WDYANYVWZ7RQ6UGZX4RQS4DXNQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.771789456 (566h29m31.642872491s) [noise=true , address=77.90.31.13:4626 , piece_id=H7Q7DP6HJJX5OGZZZ5ZS7GF6N2JQ6ABXZDIAGGJVPHM5ADT464KA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.772095738 (566h29m31.643178773s) [noise=true , address=77.90.31.11:4553 , piece_id=AFDWGXRRCGXHMDI5EYETSPXE3R7ZKJUPTQREUGIDJGY3PGWFS2SQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.772461846 (566h29m31.643544881s) [noise=true , address=145.79.227.16:4910 , piece_id=6P3OQNALH6WYHT5NGOWOLZIZLIAEDFQCZPH2WASD4UWIOJ3P5EJQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.772790912 (566h29m31.643873947s) [noise=true , address=145.79.227.4:4185 , piece_id=HOZHG57N37RJ2JK3UT7BDBIWIBOCLCU3YXFDZR34KEBOKHDYTGJA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.773220521 (566h29m31.644303556s) [noise=true , address=45.91.116.6:4974 , piece_id=7WFQU4SL32U75ONPZSVM4JLTQHGQ4HTFBBZDXLML7ZBXLKUM6WLQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.773615312 (566h29m31.644698347s) [noise=true , address=145.79.227.50:4375 , piece_id=WWTQDK7HSJG3IYIH5JQYLL7KZXM6M3URCGN6DXBBNRMYUVUGAFJA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.774075318 (566h29m31.645158353s) [noise=true , address=145.79.227.14:4840 , piece_id=YWJAWIQMLFDYCBWGFPJPAKVP3FNBQ3IS5ABRJRLWYA3IFHOSUMHQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.774396109 (566h29m31.645479144s) [noise=true , address=145.79.227.24:4582 , piece_id=CW5HR5OVSEGMSBM7VVJWUWTKJXURHOS2YRSAKLCNFUFXAZRVDXPQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.775571697 (566h29m31.646654742s) [noise=true , address=145.79.227.21:4473 , piece_id=RJERDEWEMV4LWW5PJ76FNMGBN3D3CQWJKU5KLSCM2MKVWVBFR2SA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.776659068 (566h29m31.647742103s) [noise=true , address=45.91.116.10:4081 , piece_id=KO64HRYK2IUEIIKU2N4BP2GOHOBKVTDFUHWGCPLSQAHXLWS4JC4A, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:33.455520012 (566h29m33.326603058s) [upload=496676]: Next returned nil segment Dec 29 05:58:33.455575318 (566h29m33.326658373s) [upload=496676]: Waiting for error group managing segments... Dec 29 05:58:33.751080626 (566h29m33.622163661s) [noise=true , address=145.79.227.4:4185 , piece_id=HOZHG57N37RJ2JK3UT7BDBIWIBOCLCU3YXFDZR34KEBOKHDYTGJA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.760027282 (566h29m33.631110317s) [noise=true , address=145.79.227.50:4375 , piece_id=WWTQDK7HSJG3IYIH5JQYLL7KZXM6M3URCGN6DXBBNRMYUVUGAFJA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.772617225 (566h29m33.64370027s) [noise=true , address=145.79.227.47:4266 , piece_id=M3QWVIYBOAWZ7ODPFRRMFF77DY6CSRH6ACYKS6UHY5MK47I4OCHA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.773108561 (566h29m33.644191606s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZ5VETOVJGQ46FO6BWOCKRI7C6ZQPI6A4K75F57QDAD7XOELZOQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.773528340 (566h29m33.644611375s) [noise=true , address=145.79.227.16:4910 , piece_id=6P3OQNALH6WYHT5NGOWOLZIZLIAEDFQCZPH2WASD4UWIOJ3P5EJQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.775176339 (566h29m33.646259374s) [noise=true , address=145.79.227.12:4765 , piece_id=2NV3ISJZMR2ICAYCFUGZFPRKJ2YOOC3LGFD6WMVNW6WB4ECZ4M3Q, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.784285403 (566h29m33.655368458s) [noise=true , address=145.79.227.21:4473 , piece_id=RJERDEWEMV4LWW5PJ76FNMGBN3D3CQWJKU5KLSCM2MKVWVBFR2SA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.784619810 (566h29m33.655702845s) [noise=true , address=145.79.227.14:4840 , piece_id=YWJAWIQMLFDYCBWGFPJPAKVP3FNBQ3IS5ABRJRLWYA3IFHOSUMHQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.785194175 (566h29m33.65627722s) [noise=true , address=145.79.227.24:4582 , piece_id=CW5HR5OVSEGMSBM7VVJWUWTKJXURHOS2YRSAKLCNFUFXAZRVDXPQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.786704740 (566h29m33.657787776s) [noise=true , address=145.79.227.46:4601 , piece_id=GGJTW5DEHWQ2XKDPWGDMBDM3U5WDYANYVWZ7RQ6UGZX4RQS4DXNQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.958029990 (566h29m33.829113025s) [noise=true , address=77.90.31.11:4553 , piece_id=AFDWGXRRCGXHMDI5EYETSPXE3R7ZKJUPTQREUGIDJGY3PGWFS2SQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:34.016330984 (566h29m33.887414019s) [noise=true , address=77.90.31.16:4735 , piece_id=O7FYVZSBSF44FEVF6WUH4LMAGRXM7M4JXD3BKDBEIEC4JFACMQ5Q, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:34.100715375 (566h29m33.971798411s) [noise=true , address=45.91.116.6:4974 , piece_id=7WFQU4SL32U75ONPZSVM4JLTQHGQ4HTFBBZDXLML7ZBXLKUM6WLQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:34.100741826 (566h29m33.971824861s) [seg_pos={0 0}, upload=496676]: Segment reached optimal threshold of 13 pieces. Dec 29 05:58:34.100969380 (566h29m33.972052425s) [noise=true , address=77.90.31.13:4626 , piece_id=H7Q7DP6HJJX5OGZZZ5ZS7GF6N2JQ6ABXZDIAGGJVPHM5ADT464KA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:32780->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:32780->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32780->77.90.31.13:4626: use of closed network connection Dec 29 05:58:34.100977615 (566h29m33.97206065s) [noise=true , address=77.90.31.13:4626 , piece_id=H7Q7DP6HJJX5OGZZZ5ZS7GF6N2JQ6ABXZDIAGGJVPHM5ADT464KA, seg_pos={0 0}, upload=496676]: Long tail cancellation due to optimal threshold Dec 29 05:58:34.101077516 (566h29m33.972160551s) [noise=true , address=45.91.116.10:4081 , piece_id=KO64HRYK2IUEIIKU2N4BP2GOHOBKVTDFUHWGCPLSQAHXLWS4JC4A, seg_pos={0 0}, upload=496676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:58478->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:58478->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58478->45.91.116.10:4081: use of closed network connection Dec 29 05:58:34.101093526 (566h29m33.972176561s) [noise=true , address=45.91.116.10:4081 , piece_id=KO64HRYK2IUEIIKU2N4BP2GOHOBKVTDFUHWGCPLSQAHXLWS4JC4A, seg_pos={0 0}, upload=496676]: Long tail cancellation due to optimal threshold Dec 29 05:58:34.101236468 (566h29m33.972319503s) [noise=true , address=77.90.31.19:4842 , piece_id=7V5WEYXV5ZKSUZF5Y7SWYWX7NY3K7QBECQLHEOQ5QMTNPPRAND4A, seg_pos={0 0}, upload=496676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 05:58:34.101244553 (566h29m33.972327588s) [noise=true , address=77.90.31.19:4842 , piece_id=7V5WEYXV5ZKSUZF5Y7SWYWX7NY3K7QBECQLHEOQ5QMTNPPRAND4A, seg_pos={0 0}, upload=496676]: Long tail cancellation due to optimal threshold Dec 29 05:58:34.101269110 (566h29m33.972352175s) [seg_pos={0 0}, upload=496676]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:58:34.101405570 (566h29m33.972488615s) [upload=496676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:58:34.101412783 (566h29m33.972495818s) [upload=496676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:58:34.392699891 (566h29m34.263782926s) [upload=496676]: Done uploading segments... Dec 29 05:58:34.392719026 (566h29m34.263802072s) [upload=496676]: Upload finished. err: <nil> Duration: 6.042585924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...X....................................X.X...............] duration: 15.423252886s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 06:00:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 15.423343739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...X....................................X.X..............] duration: 36.640824286s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:59:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 36.640854515s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X...X...........X.....X...X.....XX.X.....X......XX..........] duration: 4.116822264s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 05:59:46 UTC) orbiter warning: context deadline exceeded Duration: 4.116850488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X.X...XX...X.XX...........XXXX....X.X.X..X...........X.X.] duration: 5.64737107s trace: https://jaeger.dev.storj.io/trace/27e4aa47141ed4e2 FAIL (2025-12-29 05:59:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Dec 29 05:58:31.521946993 (566h29m31.393030028s) [upload=496676]: Got next segment. Inline: false Dec 29 05:58:31.522000294 (566h29m31.393083329s) [upload=496676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:58:31.522004933 (566h29m31.393087968s) [upload=496676]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:58:31.769206577 (566h29m31.640289622s) [seg_pos={0 0}, upload=496676]: Begin upload segment... Dec 29 05:58:31.769250261 (566h29m31.640333296s) [seg_pos={0 0}, upload=496676]: Done begin upload segment. Dec 29 05:58:31.769274075 (566h29m31.640357121s) [noise=true , address=145.79.227.47:4266 , piece_id=M3QWVIYBOAWZ7ODPFRRMFF77DY6CSRH6ACYKS6UHY5MK47I4OCHA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.769995219 (566h29m31.641078264s) [noise=true , address=77.90.31.19:4842 , piece_id=7V5WEYXV5ZKSUZF5Y7SWYWX7NY3K7QBECQLHEOQ5QMTNPPRAND4A, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.770470194 (566h29m31.641553239s) [noise=true , address=145.79.227.12:4765 , piece_id=2NV3ISJZMR2ICAYCFUGZFPRKJ2YOOC3LGFD6WMVNW6WB4ECZ4M3Q, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.770834607 (566h29m31.641917642s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZ5VETOVJGQ46FO6BWOCKRI7C6ZQPI6A4K75F57QDAD7XOELZOQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.771077490 (566h29m31.642160525s) [noise=true , address=77.90.31.16:4735 , piece_id=O7FYVZSBSF44FEVF6WUH4LMAGRXM7M4JXD3BKDBEIEC4JFACMQ5Q, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.771490156 (566h29m31.642573201s) [noise=true , address=145.79.227.46:4601 , piece_id=GGJTW5DEHWQ2XKDPWGDMBDM3U5WDYANYVWZ7RQ6UGZX4RQS4DXNQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.771789456 (566h29m31.642872491s) [noise=true , address=77.90.31.13:4626 , piece_id=H7Q7DP6HJJX5OGZZZ5ZS7GF6N2JQ6ABXZDIAGGJVPHM5ADT464KA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.772095738 (566h29m31.643178773s) [noise=true , address=77.90.31.11:4553 , piece_id=AFDWGXRRCGXHMDI5EYETSPXE3R7ZKJUPTQREUGIDJGY3PGWFS2SQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.772461846 (566h29m31.643544881s) [noise=true , address=145.79.227.16:4910 , piece_id=6P3OQNALH6WYHT5NGOWOLZIZLIAEDFQCZPH2WASD4UWIOJ3P5EJQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.772790912 (566h29m31.643873947s) [noise=true , address=145.79.227.4:4185 , piece_id=HOZHG57N37RJ2JK3UT7BDBIWIBOCLCU3YXFDZR34KEBOKHDYTGJA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.773220521 (566h29m31.644303556s) [noise=true , address=45.91.116.6:4974 , piece_id=7WFQU4SL32U75ONPZSVM4JLTQHGQ4HTFBBZDXLML7ZBXLKUM6WLQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.773615312 (566h29m31.644698347s) [noise=true , address=145.79.227.50:4375 , piece_id=WWTQDK7HSJG3IYIH5JQYLL7KZXM6M3URCGN6DXBBNRMYUVUGAFJA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.774075318 (566h29m31.645158353s) [noise=true , address=145.79.227.14:4840 , piece_id=YWJAWIQMLFDYCBWGFPJPAKVP3FNBQ3IS5ABRJRLWYA3IFHOSUMHQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.774396109 (566h29m31.645479144s) [noise=true , address=145.79.227.24:4582 , piece_id=CW5HR5OVSEGMSBM7VVJWUWTKJXURHOS2YRSAKLCNFUFXAZRVDXPQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.775571697 (566h29m31.646654742s) [noise=true , address=145.79.227.21:4473 , piece_id=RJERDEWEMV4LWW5PJ76FNMGBN3D3CQWJKU5KLSCM2MKVWVBFR2SA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.776659068 (566h29m31.647742103s) [noise=true , address=45.91.116.10:4081 , piece_id=KO64HRYK2IUEIIKU2N4BP2GOHOBKVTDFUHWGCPLSQAHXLWS4JC4A, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:33.455520012 (566h29m33.326603058s) [upload=496676]: Next returned nil segment Dec 29 05:58:33.455575318 (566h29m33.326658373s) [upload=496676]: Waiting for error group managing segments... Dec 29 05:58:33.751080626 (566h29m33.622163661s) [noise=true , address=145.79.227.4:4185 , piece_id=HOZHG57N37RJ2JK3UT7BDBIWIBOCLCU3YXFDZR34KEBOKHDYTGJA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.760027282 (566h29m33.631110317s) [noise=true , address=145.79.227.50:4375 , piece_id=WWTQDK7HSJG3IYIH5JQYLL7KZXM6M3URCGN6DXBBNRMYUVUGAFJA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.772617225 (566h29m33.64370027s) [noise=true , address=145.79.227.47:4266 , piece_id=M3QWVIYBOAWZ7ODPFRRMFF77DY6CSRH6ACYKS6UHY5MK47I4OCHA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.773108561 (566h29m33.644191606s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZ5VETOVJGQ46FO6BWOCKRI7C6ZQPI6A4K75F57QDAD7XOELZOQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.773528340 (566h29m33.644611375s) [noise=true , address=145.79.227.16:4910 , piece_id=6P3OQNALH6WYHT5NGOWOLZIZLIAEDFQCZPH2WASD4UWIOJ3P5EJQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.775176339 (566h29m33.646259374s) [noise=true , address=145.79.227.12:4765 , piece_id=2NV3ISJZMR2ICAYCFUGZFPRKJ2YOOC3LGFD6WMVNW6WB4ECZ4M3Q, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.784285403 (566h29m33.655368458s) [noise=true , address=145.79.227.21:4473 , piece_id=RJERDEWEMV4LWW5PJ76FNMGBN3D3CQWJKU5KLSCM2MKVWVBFR2SA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.784619810 (566h29m33.655702845s) [noise=true , address=145.79.227.14:4840 , piece_id=YWJAWIQMLFDYCBWGFPJPAKVP3FNBQ3IS5ABRJRLWYA3IFHOSUMHQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.785194175 (566h29m33.65627722s) [noise=true , address=145.79.227.24:4582 , piece_id=CW5HR5OVSEGMSBM7VVJWUWTKJXURHOS2YRSAKLCNFUFXAZRVDXPQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.786704740 (566h29m33.657787776s) [noise=true , address=145.79.227.46:4601 , piece_id=GGJTW5DEHWQ2XKDPWGDMBDM3U5WDYANYVWZ7RQ6UGZX4RQS4DXNQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.958029990 (566h29m33.829113025s) [noise=true , address=77.90.31.11:4553 , piece_id=AFDWGXRRCGXHMDI5EYETSPXE3R7ZKJUPTQREUGIDJGY3PGWFS2SQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:34.016330984 (566h29m33.887414019s) [noise=true , address=77.90.31.16:4735 , piece_id=O7FYVZSBSF44FEVF6WUH4LMAGRXM7M4JXD3BKDBEIEC4JFACMQ5Q, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:34.100715375 (566h29m33.971798411s) [noise=true , address=45.91.116.6:4974 , piece_id=7WFQU4SL32U75ONPZSVM4JLTQHGQ4HTFBBZDXLML7ZBXLKUM6WLQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:34.100741826 (566h29m33.971824861s) [seg_pos={0 0}, upload=496676]: Segment reached optimal threshold of 13 pieces. Dec 29 05:58:34.100969380 (566h29m33.972052425s) [noise=true , address=77.90.31.13:4626 , piece_id=H7Q7DP6HJJX5OGZZZ5ZS7GF6N2JQ6ABXZDIAGGJVPHM5ADT464KA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:32780->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:32780->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32780->77.90.31.13:4626: use of closed network connection Dec 29 05:58:34.100977615 (566h29m33.97206065s) [noise=true , address=77.90.31.13:4626 , piece_id=H7Q7DP6HJJX5OGZZZ5ZS7GF6N2JQ6ABXZDIAGGJVPHM5ADT464KA, seg_pos={0 0}, upload=496676]: Long tail cancellation due to optimal threshold Dec 29 05:58:34.101077516 (566h29m33.972160551s) [noise=true , address=45.91.116.10:4081 , piece_id=KO64HRYK2IUEIIKU2N4BP2GOHOBKVTDFUHWGCPLSQAHXLWS4JC4A, seg_pos={0 0}, upload=496676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:58478->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:58478->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58478->45.91.116.10:4081: use of closed network connection Dec 29 05:58:34.101093526 (566h29m33.972176561s) [noise=true , address=45.91.116.10:4081 , piece_id=KO64HRYK2IUEIIKU2N4BP2GOHOBKVTDFUHWGCPLSQAHXLWS4JC4A, seg_pos={0 0}, upload=496676]: Long tail cancellation due to optimal threshold Dec 29 05:58:34.101236468 (566h29m33.972319503s) [noise=true , address=77.90.31.19:4842 , piece_id=7V5WEYXV5ZKSUZF5Y7SWYWX7NY3K7QBECQLHEOQ5QMTNPPRAND4A, seg_pos={0 0}, upload=496676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 05:58:34.101244553 (566h29m33.972327588s) [noise=true , address=77.90.31.19:4842 , piece_id=7V5WEYXV5ZKSUZF5Y7SWYWX7NY3K7QBECQLHEOQ5QMTNPPRAND4A, seg_pos={0 0}, upload=496676]: Long tail cancellation due to optimal threshold Dec 29 05:58:34.101269110 (566h29m33.972352175s) [seg_pos={0 0}, upload=496676]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:58:34.101405570 (566h29m33.972488615s) [upload=496676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:58:34.101412783 (566h29m33.972495818s) [upload=496676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:58:34.392699891 (566h29m34.263782926s) [upload=496676]: Done uploading segments... Dec 29 05:58:34.392719026 (566h29m34.263802072s) [upload=496676]: Upload finished. err: <nil> Duration: 5.647395306s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X...X...........X.....X...X.....XX.X.....X......XX.........] duration: 3.74941786s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 05:59:09 UTC) orbiter warning: context deadline exceeded Duration: 3.749444852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X...X....................................X.X.............] duration: 21.902014338s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:59:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 21.902045668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X.X...XX...X.XX...........XXXX....X.X.X..X...........X.X] duration: 30.002774096s trace: https://jaeger.dev.storj.io/trace/27e4aa47141ed4e2 FAIL (2025-12-29 05:59:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Dec 29 05:58:31.521946993 (566h29m31.393030028s) [upload=496676]: Got next segment. Inline: false Dec 29 05:58:31.522000294 (566h29m31.393083329s) [upload=496676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:58:31.522004933 (566h29m31.393087968s) [upload=496676]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:58:31.769206577 (566h29m31.640289622s) [seg_pos={0 0}, upload=496676]: Begin upload segment... Dec 29 05:58:31.769250261 (566h29m31.640333296s) [seg_pos={0 0}, upload=496676]: Done begin upload segment. Dec 29 05:58:31.769274075 (566h29m31.640357121s) [noise=true , address=145.79.227.47:4266 , piece_id=M3QWVIYBOAWZ7ODPFRRMFF77DY6CSRH6ACYKS6UHY5MK47I4OCHA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.769995219 (566h29m31.641078264s) [noise=true , address=77.90.31.19:4842 , piece_id=7V5WEYXV5ZKSUZF5Y7SWYWX7NY3K7QBECQLHEOQ5QMTNPPRAND4A, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.770470194 (566h29m31.641553239s) [noise=true , address=145.79.227.12:4765 , piece_id=2NV3ISJZMR2ICAYCFUGZFPRKJ2YOOC3LGFD6WMVNW6WB4ECZ4M3Q, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.770834607 (566h29m31.641917642s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZ5VETOVJGQ46FO6BWOCKRI7C6ZQPI6A4K75F57QDAD7XOELZOQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.771077490 (566h29m31.642160525s) [noise=true , address=77.90.31.16:4735 , piece_id=O7FYVZSBSF44FEVF6WUH4LMAGRXM7M4JXD3BKDBEIEC4JFACMQ5Q, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.771490156 (566h29m31.642573201s) [noise=true , address=145.79.227.46:4601 , piece_id=GGJTW5DEHWQ2XKDPWGDMBDM3U5WDYANYVWZ7RQ6UGZX4RQS4DXNQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.771789456 (566h29m31.642872491s) [noise=true , address=77.90.31.13:4626 , piece_id=H7Q7DP6HJJX5OGZZZ5ZS7GF6N2JQ6ABXZDIAGGJVPHM5ADT464KA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.772095738 (566h29m31.643178773s) [noise=true , address=77.90.31.11:4553 , piece_id=AFDWGXRRCGXHMDI5EYETSPXE3R7ZKJUPTQREUGIDJGY3PGWFS2SQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.772461846 (566h29m31.643544881s) [noise=true , address=145.79.227.16:4910 , piece_id=6P3OQNALH6WYHT5NGOWOLZIZLIAEDFQCZPH2WASD4UWIOJ3P5EJQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.772790912 (566h29m31.643873947s) [noise=true , address=145.79.227.4:4185 , piece_id=HOZHG57N37RJ2JK3UT7BDBIWIBOCLCU3YXFDZR34KEBOKHDYTGJA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.773220521 (566h29m31.644303556s) [noise=true , address=45.91.116.6:4974 , piece_id=7WFQU4SL32U75ONPZSVM4JLTQHGQ4HTFBBZDXLML7ZBXLKUM6WLQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.773615312 (566h29m31.644698347s) [noise=true , address=145.79.227.50:4375 , piece_id=WWTQDK7HSJG3IYIH5JQYLL7KZXM6M3URCGN6DXBBNRMYUVUGAFJA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.774075318 (566h29m31.645158353s) [noise=true , address=145.79.227.14:4840 , piece_id=YWJAWIQMLFDYCBWGFPJPAKVP3FNBQ3IS5ABRJRLWYA3IFHOSUMHQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.774396109 (566h29m31.645479144s) [noise=true , address=145.79.227.24:4582 , piece_id=CW5HR5OVSEGMSBM7VVJWUWTKJXURHOS2YRSAKLCNFUFXAZRVDXPQ, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.775571697 (566h29m31.646654742s) [noise=true , address=145.79.227.21:4473 , piece_id=RJERDEWEMV4LWW5PJ76FNMGBN3D3CQWJKU5KLSCM2MKVWVBFR2SA, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:31.776659068 (566h29m31.647742103s) [noise=true , address=45.91.116.10:4081 , piece_id=KO64HRYK2IUEIIKU2N4BP2GOHOBKVTDFUHWGCPLSQAHXLWS4JC4A, seg_pos={0 0}, upload=496676]: Uploading piece... Dec 29 05:58:33.455520012 (566h29m33.326603058s) [upload=496676]: Next returned nil segment Dec 29 05:58:33.455575318 (566h29m33.326658373s) [upload=496676]: Waiting for error group managing segments... Dec 29 05:58:33.751080626 (566h29m33.622163661s) [noise=true , address=145.79.227.4:4185 , piece_id=HOZHG57N37RJ2JK3UT7BDBIWIBOCLCU3YXFDZR34KEBOKHDYTGJA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.760027282 (566h29m33.631110317s) [noise=true , address=145.79.227.50:4375 , piece_id=WWTQDK7HSJG3IYIH5JQYLL7KZXM6M3URCGN6DXBBNRMYUVUGAFJA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.772617225 (566h29m33.64370027s) [noise=true , address=145.79.227.47:4266 , piece_id=M3QWVIYBOAWZ7ODPFRRMFF77DY6CSRH6ACYKS6UHY5MK47I4OCHA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.773108561 (566h29m33.644191606s) [noise=true , address=145.79.227.38:4978 , piece_id=ICZ5VETOVJGQ46FO6BWOCKRI7C6ZQPI6A4K75F57QDAD7XOELZOQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.773528340 (566h29m33.644611375s) [noise=true , address=145.79.227.16:4910 , piece_id=6P3OQNALH6WYHT5NGOWOLZIZLIAEDFQCZPH2WASD4UWIOJ3P5EJQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.775176339 (566h29m33.646259374s) [noise=true , address=145.79.227.12:4765 , piece_id=2NV3ISJZMR2ICAYCFUGZFPRKJ2YOOC3LGFD6WMVNW6WB4ECZ4M3Q, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.784285403 (566h29m33.655368458s) [noise=true , address=145.79.227.21:4473 , piece_id=RJERDEWEMV4LWW5PJ76FNMGBN3D3CQWJKU5KLSCM2MKVWVBFR2SA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.784619810 (566h29m33.655702845s) [noise=true , address=145.79.227.14:4840 , piece_id=YWJAWIQMLFDYCBWGFPJPAKVP3FNBQ3IS5ABRJRLWYA3IFHOSUMHQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.785194175 (566h29m33.65627722s) [noise=true , address=145.79.227.24:4582 , piece_id=CW5HR5OVSEGMSBM7VVJWUWTKJXURHOS2YRSAKLCNFUFXAZRVDXPQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.786704740 (566h29m33.657787776s) [noise=true , address=145.79.227.46:4601 , piece_id=GGJTW5DEHWQ2XKDPWGDMBDM3U5WDYANYVWZ7RQ6UGZX4RQS4DXNQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:33.958029990 (566h29m33.829113025s) [noise=true , address=77.90.31.11:4553 , piece_id=AFDWGXRRCGXHMDI5EYETSPXE3R7ZKJUPTQREUGIDJGY3PGWFS2SQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:34.016330984 (566h29m33.887414019s) [noise=true , address=77.90.31.16:4735 , piece_id=O7FYVZSBSF44FEVF6WUH4LMAGRXM7M4JXD3BKDBEIEC4JFACMQ5Q, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:34.100715375 (566h29m33.971798411s) [noise=true , address=45.91.116.6:4974 , piece_id=7WFQU4SL32U75ONPZSVM4JLTQHGQ4HTFBBZDXLML7ZBXLKUM6WLQ, seg_pos={0 0}, upload=496676]: Done uploading piece. err: <nil> Dec 29 05:58:34.100741826 (566h29m33.971824861s) [seg_pos={0 0}, upload=496676]: Segment reached optimal threshold of 13 pieces. Dec 29 05:58:34.100969380 (566h29m33.972052425s) [noise=true , address=77.90.31.13:4626 , piece_id=H7Q7DP6HJJX5OGZZZ5ZS7GF6N2JQ6ABXZDIAGGJVPHM5ADT464KA, seg_pos={0 0}, upload=496676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:32780->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:32780->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32780->77.90.31.13:4626: use of closed network connection Dec 29 05:58:34.100977615 (566h29m33.97206065s) [noise=true , address=77.90.31.13:4626 , piece_id=H7Q7DP6HJJX5OGZZZ5ZS7GF6N2JQ6ABXZDIAGGJVPHM5ADT464KA, seg_pos={0 0}, upload=496676]: Long tail cancellation due to optimal threshold Dec 29 05:58:34.101077516 (566h29m33.972160551s) [noise=true , address=45.91.116.10:4081 , piece_id=KO64HRYK2IUEIIKU2N4BP2GOHOBKVTDFUHWGCPLSQAHXLWS4JC4A, seg_pos={0 0}, upload=496676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:58478->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:58478->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58478->45.91.116.10:4081: use of closed network connection Dec 29 05:58:34.101093526 (566h29m33.972176561s) [noise=true , address=45.91.116.10:4081 , piece_id=KO64HRYK2IUEIIKU2N4BP2GOHOBKVTDFUHWGCPLSQAHXLWS4JC4A, seg_pos={0 0}, upload=496676]: Long tail cancellation due to optimal threshold Dec 29 05:58:34.101236468 (566h29m33.972319503s) [noise=true , address=77.90.31.19:4842 , piece_id=7V5WEYXV5ZKSUZF5Y7SWYWX7NY3K7QBECQLHEOQ5QMTNPPRAND4A, seg_pos={0 0}, upload=496676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 05:58:34.101244553 (566h29m33.972327588s) [noise=true , address=77.90.31.19:4842 , piece_id=7V5WEYXV5ZKSUZF5Y7SWYWX7NY3K7QBECQLHEOQ5QMTNPPRAND4A, seg_pos={0 0}, upload=496676]: Long tail cancellation due to optimal threshold Dec 29 05:58:34.101269110 (566h29m33.972352175s) [seg_pos={0 0}, upload=496676]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:58:34.101405570 (566h29m33.972488615s) [upload=496676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:58:34.101412783 (566h29m33.972495818s) [upload=496676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:58:34.392699891 (566h29m34.263782926s) [upload=496676]: Done uploading segments... Dec 29 05:58:34.392719026 (566h29m34.263802072s) [upload=496676]: Upload finished. err: <nil> Duration: 30.002802361s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X...X....................................X.X............] duration: 15.396834412s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:58:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 15.396863988s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..X...X...........X.....X...X.....XX.X.....X......XX........] duration: 3.465393512s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 05:58:22 UTC) orbiter warning: context deadline exceeded Duration: 3.465409003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X...X....................................X.X...........] duration: 34.269555725s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:58:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 34.269587726s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...X...X...........X.....X...X.....XX.X.....X......XX.......] duration: 3.609873345s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 05:57:56 UTC) orbiter warning: context deadline exceeded Duration: 3.609893113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X..X.X...XX...X.XX...........XXXX....X.X.X..X...........X.] duration: 5.94832747s trace: https://jaeger.dev.storj.io/trace/4f1e150242c0c070 FAIL (2025-12-29 05:57:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 4WHGOS4B6OG3PIP2XY4DDQSMGC3RNY35OJ5TXBARNOMNGWPB23ZQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: Q4AAO72XX3IGCJQHMDMZSDPMHEKKBO5CJG3DKGS5B56HQ6CAHHNQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: K4OW7PCVGYXFKBSJIT5YFETK4FKGRE53WBYYUUYNAV3ZLARLPEAQ) context deadline exceeded ======LOGS====== :56:30.186436846 (566h27m30.057519891s) [upload=496647]: Starting upload Dec 29 05:56:30.186498043 (566h27m30.057581108s) [upload=496647]: Done starting upload Dec 29 05:56:30.186685690 (566h27m30.057768725s) [upload=496647]: Uploading segments... Dec 29 05:56:30.186783657 (566h27m30.057866732s) [upload=496647]: Got next segment. Inline: false Dec 29 05:56:30.186874539 (566h27m30.057957584s) [upload=496647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:56:30.186881193 (566h27m30.057964238s) [upload=496647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:56:30.428880666 (566h27m30.299963712s) [seg_pos={0 0}, upload=496647]: Begin upload segment... Dec 29 05:56:30.428934690 (566h27m30.300017725s) [seg_pos={0 0}, upload=496647]: Done begin upload segment. Dec 29 05:56:30.428959887 (566h27m30.300042932s) [noise=true , address=145.79.227.12:4765 , piece_id=LERBDBOSPJSBUY4G2FL3SXZMQAJICXMYVC7QWLYOP5VAPDTF2YMA, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.428991648 (566h27m30.300074683s) [noise=true , address=145.79.227.39:4015 , piece_id=WWCF6GYGXGMTUMX75YCZGDGJJWD7SJOJ5H5MI4KWQR6TNQDDPISQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.429495097 (566h27m30.300578142s) [noise=true , address=77.90.31.16:4735 , piece_id=4WHGOS4B6OG3PIP2XY4DDQSMGC3RNY35OJ5TXBARNOMNGWPB23ZQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.429791931 (566h27m30.300874977s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4AAO72XX3IGCJQHMDMZSDPMHEKKBO5CJG3DKGS5B56HQ6CAHHNQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.430283207 (566h27m30.301366242s) [noise=true , address=145.79.227.16:4910 , piece_id=AE4GMVQYTJSOXGZYGZ6A7VLZMIZE6LGMIVIUGKOB5QNEMMLBSOSQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.430618987 (566h27m30.301702032s) [noise=true , address=145.79.227.45:4229 , piece_id=3H54ZKNHUOONE6WFH6U2GBBVFDG5AB4JPI6VL47S6CVBEAX7B5OQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.430911894 (566h27m30.301994939s) [noise=true , address=145.79.227.5:4223 , piece_id=LDVL6MVOUC6ZYLCDYS4FQKUWC64P3TLLD6MDCDLBJTM34VHOHU6A, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.431192308 (566h27m30.302275343s) [noise=true , address=45.91.116.12:4228 , piece_id=FOUVZ2APE24PXFX4INQVKMHP3BFNU5WVWMCY3H3DWBCQDJPV5ECQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.431539118 (566h27m30.302622163s) [noise=true , address=145.79.227.38:4979 , piece_id=JI4AWT575COD6HWF7QOA6R22YNCOT32UN2HL67XJ465UHGMTPV7Q, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.433061628 (566h27m30.304144663s) [noise=true , address=45.91.116.5:4300 , piece_id=K4OW7PCVGYXFKBSJIT5YFETK4FKGRE53WBYYUUYNAV3ZLARLPEAQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.433443585 (566h27m30.30452663s) [noise=true , address=145.79.227.4:4186 , piece_id=2LNNIS47ETBEJP3CYIXJU2RIYWIOFWZY7HRCCBBQMV5B7GIAGU4A, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.433792579 (566h27m30.304875615s) [noise=true , address=145.79.227.7:4295 , piece_id=JZD45H24GPZOXZ3GWDNARK2DVHPKYYGLYVNML7KECNAFPGZVKXCQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.435631261 (566h27m30.306714326s) [noise=true , address=45.91.116.14:4191 , piece_id=4QT63GNUTEEQLDICT7A5O32KUXAASB5MOPQUFK6SMCFIXMBLJ2EA, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.436211807 (566h27m30.307294842s) [noise=true , address=45.91.116.6:4976 , piece_id=UQ5HJRQKYAXR6D5WQCKG4GTCDUA3BGC3TNODNP7LEMF5OC6UNAWQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.437497093 (566h27m30.308580139s) [noise=true , address=145.79.227.24:4582 , piece_id=XLUEHTORFZ5U7SEBNKMO5HK4M6BTY4X7GG7HYRCB3HNLLGA36A3Q, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.438014529 (566h27m30.309097564s) [noise=true , address=77.90.31.14:4661 , piece_id=FAIH2CBQZAMOZABMCKMSQQCACACK5AB6FOSOXLLLWPO4BKCW3NQQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:32.030391827 (566h27m31.901474862s) [upload=496647]: Next returned nil segment Dec 29 05:56:32.030404171 (566h27m31.901487206s) [upload=496647]: Waiting for error group managing segments... Dec 29 05:56:32.413483183 (566h27m32.284566228s) [noise=true , address=145.79.227.12:4765 , piece_id=LERBDBOSPJSBUY4G2FL3SXZMQAJICXMYVC7QWLYOP5VAPDTF2YMA, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.414715470 (566h27m32.285798515s) [noise=true , address=145.79.227.5:4223 , piece_id=LDVL6MVOUC6ZYLCDYS4FQKUWC64P3TLLD6MDCDLBJTM34VHOHU6A, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.414962841 (566h27m32.286045876s) [noise=true , address=145.79.227.7:4295 , piece_id=JZD45H24GPZOXZ3GWDNARK2DVHPKYYGLYVNML7KECNAFPGZVKXCQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.420367602 (566h27m32.291450637s) [noise=true , address=145.79.227.4:4186 , piece_id=2LNNIS47ETBEJP3CYIXJU2RIYWIOFWZY7HRCCBBQMV5B7GIAGU4A, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.421574159 (566h27m32.292657194s) [noise=true , address=145.79.227.24:4582 , piece_id=XLUEHTORFZ5U7SEBNKMO5HK4M6BTY4X7GG7HYRCB3HNLLGA36A3Q, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.422057259 (566h27m32.293140294s) [noise=true , address=145.79.227.39:4015 , piece_id=WWCF6GYGXGMTUMX75YCZGDGJJWD7SJOJ5H5MI4KWQR6TNQDDPISQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.430548889 (566h27m32.301631924s) [noise=true , address=145.79.227.16:4910 , piece_id=AE4GMVQYTJSOXGZYGZ6A7VLZMIZE6LGMIVIUGKOB5QNEMMLBSOSQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.441203717 (566h27m32.312286763s) [noise=true , address=145.79.227.45:4229 , piece_id=3H54ZKNHUOONE6WFH6U2GBBVFDG5AB4JPI6VL47S6CVBEAX7B5OQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.444851694 (566h27m32.315934739s) [noise=true , address=145.79.227.38:4979 , piece_id=JI4AWT575COD6HWF7QOA6R22YNCOT32UN2HL67XJ465UHGMTPV7Q, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.683766926 (566h27m32.554849961s) [noise=true , address=45.91.116.14:4191 , piece_id=4QT63GNUTEEQLDICT7A5O32KUXAASB5MOPQUFK6SMCFIXMBLJ2EA, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.798682744 (566h27m32.669765789s) [noise=true , address=77.90.31.16:4735 , piece_id=4WHGOS4B6OG3PIP2XY4DDQSMGC3RNY35OJ5TXBARNOMNGWPB23ZQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.865539800 (566h27m32.736622845s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4AAO72XX3IGCJQHMDMZSDPMHEKKBO5CJG3DKGS5B56HQ6CAHHNQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:33.106664790 (566h27m32.977747835s) [noise=true , address=45.91.116.5:4300 , piece_id=K4OW7PCVGYXFKBSJIT5YFETK4FKGRE53WBYYUUYNAV3ZLARLPEAQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:33.106689517 (566h27m32.977772552s) [seg_pos={0 0}, upload=496647]: Segment reached optimal threshold of 13 pieces. Dec 29 05:56:33.106923863 (566h27m32.978006898s) [noise=true , address=45.91.116.6:4976 , piece_id=UQ5HJRQKYAXR6D5WQCKG4GTCDUA3BGC3TNODNP7LEMF5OC6UNAWQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 29 05:56:33.106934353 (566h27m32.978017388s) [noise=true , address=45.91.116.6:4976 , piece_id=UQ5HJRQKYAXR6D5WQCKG4GTCDUA3BGC3TNODNP7LEMF5OC6UNAWQ, seg_pos={0 0}, upload=496647]: Long tail cancellation due to optimal threshold Dec 29 05:56:33.106967757 (566h27m32.978050792s) [noise=true , address=45.91.116.12:4228 , piece_id=FOUVZ2APE24PXFX4INQVKMHP3BFNU5WVWMCY3H3DWBCQDJPV5ECQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:52310->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:52310->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52310->45.91.116.12:4228: use of closed network connection Dec 29 05:56:33.106975431 (566h27m32.978058466s) [noise=true , address=45.91.116.12:4228 , piece_id=FOUVZ2APE24PXFX4INQVKMHP3BFNU5WVWMCY3H3DWBCQDJPV5ECQ, seg_pos={0 0}, upload=496647]: Long tail cancellation due to optimal threshold Dec 29 05:56:33.107239795 (566h27m32.97832285s) [noise=true , address=77.90.31.14:4661 , piece_id=FAIH2CBQZAMOZABMCKMSQQCACACK5AB6FOSOXLLLWPO4BKCW3NQQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 05:56:33.107248862 (566h27m32.978331907s) [noise=true , address=77.90.31.14:4661 , piece_id=FAIH2CBQZAMOZABMCKMSQQCACACK5AB6FOSOXLLLWPO4BKCW3NQQ, seg_pos={0 0}, upload=496647]: Long tail cancellation due to optimal threshold Dec 29 05:56:33.107292886 (566h27m32.978375931s) [seg_pos={0 0}, upload=496647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:56:33.107409447 (566h27m32.978492502s) [upload=496647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:56:33.107416081 (566h27m32.978499136s) [upload=496647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:56:33.390730285 (566h27m33.26181332s) [upload=496647]: Done uploading segments... Dec 29 05:56:33.390749462 (566h27m33.261832507s) [upload=496647]: Upload finished. err: <nil> Duration: 5.948354401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..X...X....................................X.X..........] duration: 16.156783523s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:57:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 16.156799675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X..X.X...XX...X.XX...........XXXX....X.X.X..X...........X] duration: 30.003853372s trace: https://jaeger.dev.storj.io/trace/4f1e150242c0c070 FAIL (2025-12-29 05:56:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 4WHGOS4B6OG3PIP2XY4DDQSMGC3RNY35OJ5TXBARNOMNGWPB23ZQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: Q4AAO72XX3IGCJQHMDMZSDPMHEKKBO5CJG3DKGS5B56HQ6CAHHNQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: K4OW7PCVGYXFKBSJIT5YFETK4FKGRE53WBYYUUYNAV3ZLARLPEAQ) context deadline exceeded ======LOGS====== :56:30.186436846 (566h27m30.057519891s) [upload=496647]: Starting upload Dec 29 05:56:30.186498043 (566h27m30.057581108s) [upload=496647]: Done starting upload Dec 29 05:56:30.186685690 (566h27m30.057768725s) [upload=496647]: Uploading segments... Dec 29 05:56:30.186783657 (566h27m30.057866732s) [upload=496647]: Got next segment. Inline: false Dec 29 05:56:30.186874539 (566h27m30.057957584s) [upload=496647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:56:30.186881193 (566h27m30.057964238s) [upload=496647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:56:30.428880666 (566h27m30.299963712s) [seg_pos={0 0}, upload=496647]: Begin upload segment... Dec 29 05:56:30.428934690 (566h27m30.300017725s) [seg_pos={0 0}, upload=496647]: Done begin upload segment. Dec 29 05:56:30.428959887 (566h27m30.300042932s) [noise=true , address=145.79.227.12:4765 , piece_id=LERBDBOSPJSBUY4G2FL3SXZMQAJICXMYVC7QWLYOP5VAPDTF2YMA, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.428991648 (566h27m30.300074683s) [noise=true , address=145.79.227.39:4015 , piece_id=WWCF6GYGXGMTUMX75YCZGDGJJWD7SJOJ5H5MI4KWQR6TNQDDPISQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.429495097 (566h27m30.300578142s) [noise=true , address=77.90.31.16:4735 , piece_id=4WHGOS4B6OG3PIP2XY4DDQSMGC3RNY35OJ5TXBARNOMNGWPB23ZQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.429791931 (566h27m30.300874977s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4AAO72XX3IGCJQHMDMZSDPMHEKKBO5CJG3DKGS5B56HQ6CAHHNQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.430283207 (566h27m30.301366242s) [noise=true , address=145.79.227.16:4910 , piece_id=AE4GMVQYTJSOXGZYGZ6A7VLZMIZE6LGMIVIUGKOB5QNEMMLBSOSQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.430618987 (566h27m30.301702032s) [noise=true , address=145.79.227.45:4229 , piece_id=3H54ZKNHUOONE6WFH6U2GBBVFDG5AB4JPI6VL47S6CVBEAX7B5OQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.430911894 (566h27m30.301994939s) [noise=true , address=145.79.227.5:4223 , piece_id=LDVL6MVOUC6ZYLCDYS4FQKUWC64P3TLLD6MDCDLBJTM34VHOHU6A, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.431192308 (566h27m30.302275343s) [noise=true , address=45.91.116.12:4228 , piece_id=FOUVZ2APE24PXFX4INQVKMHP3BFNU5WVWMCY3H3DWBCQDJPV5ECQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.431539118 (566h27m30.302622163s) [noise=true , address=145.79.227.38:4979 , piece_id=JI4AWT575COD6HWF7QOA6R22YNCOT32UN2HL67XJ465UHGMTPV7Q, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.433061628 (566h27m30.304144663s) [noise=true , address=45.91.116.5:4300 , piece_id=K4OW7PCVGYXFKBSJIT5YFETK4FKGRE53WBYYUUYNAV3ZLARLPEAQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.433443585 (566h27m30.30452663s) [noise=true , address=145.79.227.4:4186 , piece_id=2LNNIS47ETBEJP3CYIXJU2RIYWIOFWZY7HRCCBBQMV5B7GIAGU4A, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.433792579 (566h27m30.304875615s) [noise=true , address=145.79.227.7:4295 , piece_id=JZD45H24GPZOXZ3GWDNARK2DVHPKYYGLYVNML7KECNAFPGZVKXCQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.435631261 (566h27m30.306714326s) [noise=true , address=45.91.116.14:4191 , piece_id=4QT63GNUTEEQLDICT7A5O32KUXAASB5MOPQUFK6SMCFIXMBLJ2EA, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.436211807 (566h27m30.307294842s) [noise=true , address=45.91.116.6:4976 , piece_id=UQ5HJRQKYAXR6D5WQCKG4GTCDUA3BGC3TNODNP7LEMF5OC6UNAWQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.437497093 (566h27m30.308580139s) [noise=true , address=145.79.227.24:4582 , piece_id=XLUEHTORFZ5U7SEBNKMO5HK4M6BTY4X7GG7HYRCB3HNLLGA36A3Q, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:30.438014529 (566h27m30.309097564s) [noise=true , address=77.90.31.14:4661 , piece_id=FAIH2CBQZAMOZABMCKMSQQCACACK5AB6FOSOXLLLWPO4BKCW3NQQ, seg_pos={0 0}, upload=496647]: Uploading piece... Dec 29 05:56:32.030391827 (566h27m31.901474862s) [upload=496647]: Next returned nil segment Dec 29 05:56:32.030404171 (566h27m31.901487206s) [upload=496647]: Waiting for error group managing segments... Dec 29 05:56:32.413483183 (566h27m32.284566228s) [noise=true , address=145.79.227.12:4765 , piece_id=LERBDBOSPJSBUY4G2FL3SXZMQAJICXMYVC7QWLYOP5VAPDTF2YMA, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.414715470 (566h27m32.285798515s) [noise=true , address=145.79.227.5:4223 , piece_id=LDVL6MVOUC6ZYLCDYS4FQKUWC64P3TLLD6MDCDLBJTM34VHOHU6A, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.414962841 (566h27m32.286045876s) [noise=true , address=145.79.227.7:4295 , piece_id=JZD45H24GPZOXZ3GWDNARK2DVHPKYYGLYVNML7KECNAFPGZVKXCQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.420367602 (566h27m32.291450637s) [noise=true , address=145.79.227.4:4186 , piece_id=2LNNIS47ETBEJP3CYIXJU2RIYWIOFWZY7HRCCBBQMV5B7GIAGU4A, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.421574159 (566h27m32.292657194s) [noise=true , address=145.79.227.24:4582 , piece_id=XLUEHTORFZ5U7SEBNKMO5HK4M6BTY4X7GG7HYRCB3HNLLGA36A3Q, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.422057259 (566h27m32.293140294s) [noise=true , address=145.79.227.39:4015 , piece_id=WWCF6GYGXGMTUMX75YCZGDGJJWD7SJOJ5H5MI4KWQR6TNQDDPISQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.430548889 (566h27m32.301631924s) [noise=true , address=145.79.227.16:4910 , piece_id=AE4GMVQYTJSOXGZYGZ6A7VLZMIZE6LGMIVIUGKOB5QNEMMLBSOSQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.441203717 (566h27m32.312286763s) [noise=true , address=145.79.227.45:4229 , piece_id=3H54ZKNHUOONE6WFH6U2GBBVFDG5AB4JPI6VL47S6CVBEAX7B5OQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.444851694 (566h27m32.315934739s) [noise=true , address=145.79.227.38:4979 , piece_id=JI4AWT575COD6HWF7QOA6R22YNCOT32UN2HL67XJ465UHGMTPV7Q, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.683766926 (566h27m32.554849961s) [noise=true , address=45.91.116.14:4191 , piece_id=4QT63GNUTEEQLDICT7A5O32KUXAASB5MOPQUFK6SMCFIXMBLJ2EA, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.798682744 (566h27m32.669765789s) [noise=true , address=77.90.31.16:4735 , piece_id=4WHGOS4B6OG3PIP2XY4DDQSMGC3RNY35OJ5TXBARNOMNGWPB23ZQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:32.865539800 (566h27m32.736622845s) [noise=true , address=77.90.31.13:4626 , piece_id=Q4AAO72XX3IGCJQHMDMZSDPMHEKKBO5CJG3DKGS5B56HQ6CAHHNQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:33.106664790 (566h27m32.977747835s) [noise=true , address=45.91.116.5:4300 , piece_id=K4OW7PCVGYXFKBSJIT5YFETK4FKGRE53WBYYUUYNAV3ZLARLPEAQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: <nil> Dec 29 05:56:33.106689517 (566h27m32.977772552s) [seg_pos={0 0}, upload=496647]: Segment reached optimal threshold of 13 pieces. Dec 29 05:56:33.106923863 (566h27m32.978006898s) [noise=true , address=45.91.116.6:4976 , piece_id=UQ5HJRQKYAXR6D5WQCKG4GTCDUA3BGC3TNODNP7LEMF5OC6UNAWQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 29 05:56:33.106934353 (566h27m32.978017388s) [noise=true , address=45.91.116.6:4976 , piece_id=UQ5HJRQKYAXR6D5WQCKG4GTCDUA3BGC3TNODNP7LEMF5OC6UNAWQ, seg_pos={0 0}, upload=496647]: Long tail cancellation due to optimal threshold Dec 29 05:56:33.106967757 (566h27m32.978050792s) [noise=true , address=45.91.116.12:4228 , piece_id=FOUVZ2APE24PXFX4INQVKMHP3BFNU5WVWMCY3H3DWBCQDJPV5ECQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:52310->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:52310->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52310->45.91.116.12:4228: use of closed network connection Dec 29 05:56:33.106975431 (566h27m32.978058466s) [noise=true , address=45.91.116.12:4228 , piece_id=FOUVZ2APE24PXFX4INQVKMHP3BFNU5WVWMCY3H3DWBCQDJPV5ECQ, seg_pos={0 0}, upload=496647]: Long tail cancellation due to optimal threshold Dec 29 05:56:33.107239795 (566h27m32.97832285s) [noise=true , address=77.90.31.14:4661 , piece_id=FAIH2CBQZAMOZABMCKMSQQCACACK5AB6FOSOXLLLWPO4BKCW3NQQ, seg_pos={0 0}, upload=496647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 05:56:33.107248862 (566h27m32.978331907s) [noise=true , address=77.90.31.14:4661 , piece_id=FAIH2CBQZAMOZABMCKMSQQCACACK5AB6FOSOXLLLWPO4BKCW3NQQ, seg_pos={0 0}, upload=496647]: Long tail cancellation due to optimal threshold Dec 29 05:56:33.107292886 (566h27m32.978375931s) [seg_pos={0 0}, upload=496647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:56:33.107409447 (566h27m32.978492502s) [upload=496647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:56:33.107416081 (566h27m32.978499136s) [upload=496647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:56:33.390730285 (566h27m33.26181332s) [upload=496647]: Done uploading segments... Dec 29 05:56:33.390749462 (566h27m33.261832507s) [upload=496647]: Upload finished. err: <nil> Duration: 30.003877289s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..X...X....................................X.X.........] duration: 28.204698319s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:56:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 28.204728165s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [....X...X...........X.....X...X.....XX.X.....X......XX......] duration: 3.473322142s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 05:56:56 UTC) orbiter warning: context deadline exceeded Duration: 3.473356027s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.....X...X...........X.....X...X.....XX.X.....X......XX.....] duration: 3.367230344s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 05:56:38 UTC) orbiter warning: context deadline exceeded Duration: 3.36728071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..X...X....................................X.X........] duration: 15.419506672s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:56:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 15.419541028s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [......X...X...........X.....X...X.....XX.X.....X......XX....] duration: 4.336794007s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 05:56:23 UTC) orbiter warning: context deadline exceeded Duration: 4.336814266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.....X..X...X....................................X.X.......] duration: 33.024182376s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:56:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 33.024198126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.....X..X...X....................................X.X......] duration: 34.22304848s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:55:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 34.22307952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X..X.X...XX...X.XX...........XXXX....X.X.X..X...........] duration: 5.968974996s trace: https://jaeger.dev.storj.io/trace/7c4c54f5bda64ae7 FAIL (2025-12-29 05:55:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 05:47:06.139220037 (566h18m6.010303092s) [upload=496515]: Starting upload Dec 29 05:47:06.139302664 (566h18m6.010385709s) [upload=496515]: Done starting upload Dec 29 05:47:06.139471035 (566h18m6.01055409s) [upload=496515]: Uploading segments... Dec 29 05:47:06.139568380 (566h18m6.010651435s) [upload=496515]: Got next segment. Inline: false Dec 29 05:47:06.139620329 (566h18m6.010703364s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:47:06.139626592 (566h18m6.010709637s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:47:06.407587957 (566h18m6.278670992s) [seg_pos={0 0}, upload=496515]: Begin upload segment... Dec 29 05:47:06.407654203 (566h18m6.278737238s) [seg_pos={0 0}, upload=496515]: Done begin upload segment. Dec 29 05:47:06.407687316 (566h18m6.278770351s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407739155 (566h18m6.27882219s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407836391 (566h18m6.278919426s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408219260 (566h18m6.279302295s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408593622 (566h18m6.279676657s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409254480 (566h18m6.280337515s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409689258 (566h18m6.280772294s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410118516 (566h18m6.281201561s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410504250 (566h18m6.281587295s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.411968288 (566h18m6.283051333s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.413176318 (566h18m6.284259363s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.414893808 (566h18m6.285976853s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416341014 (566h18m6.28742405s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416677855 (566h18m6.28776089s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416696692 (566h18m6.287779737s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416776694 (566h18m6.287859729s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:08.105624686 (566h18m7.976707731s) [upload=496515]: Next returned nil segment Dec 29 05:47:08.105645005 (566h18m7.97672805s) [upload=496515]: Waiting for error group managing segments... Dec 29 05:47:08.387857626 (566h18m8.258940671s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398533746 (566h18m8.269616791s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398556088 (566h18m8.269639133s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.400797887 (566h18m8.271880932s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.403560297 (566h18m8.274643342s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.405882128 (566h18m8.276965173s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.408808900 (566h18m8.279891945s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.419006578 (566h18m8.290089624s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.421649471 (566h18m8.292732516s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.422123274 (566h18m8.293206319s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.423940313 (566h18m8.295023348s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.656954976 (566h18m8.528038011s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725071939 (566h18m8.596154985s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725097759 (566h18m8.596180804s) [seg_pos={0 0}, upload=496515]: Segment reached optimal threshold of 13 pieces. Dec 29 05:47:08.725351091 (566h18m8.596434136s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725361110 (566h18m8.596444165s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725504133 (566h18m8.596587208s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725514573 (566h18m8.596597618s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725713051 (566h18m8.596796096s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725722368 (566h18m8.596805413s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725743438 (566h18m8.596826483s) [seg_pos={0 0}, upload=496515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:47:08.725873205 (566h18m8.59695626s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:47:08.725881431 (566h18m8.596964476s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:47:09.207281964 (566h18m9.078365009s) [upload=496515]: Done uploading segments... Dec 29 05:47:09.207379881 (566h18m9.078462926s) [upload=496515]: Upload finished. err: <nil> Duration: 5.969001818s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.......X...X...........X.....X...X.....XX.X.....X......XX...] duration: 4.446961904s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 05:55:24 UTC) orbiter warning: context deadline exceeded Duration: 4.446987994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.....X..X...X....................................X.X.....] duration: 50.327603429s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:54:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 50.327632454s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [........X...X...........X.....X...X.....XX.X.....X......XX..] duration: 5.000504772s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 05:54:37 UTC) orbiter warning: context deadline exceeded Duration: 5.000520281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X..X.X...XX...X.XX...........XXXX....X.X.X..X..........] duration: 5.947884567s trace: https://jaeger.dev.storj.io/trace/7c4c54f5bda64ae7 FAIL (2025-12-29 05:54:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 05:47:06.139220037 (566h18m6.010303092s) [upload=496515]: Starting upload Dec 29 05:47:06.139302664 (566h18m6.010385709s) [upload=496515]: Done starting upload Dec 29 05:47:06.139471035 (566h18m6.01055409s) [upload=496515]: Uploading segments... Dec 29 05:47:06.139568380 (566h18m6.010651435s) [upload=496515]: Got next segment. Inline: false Dec 29 05:47:06.139620329 (566h18m6.010703364s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:47:06.139626592 (566h18m6.010709637s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:47:06.407587957 (566h18m6.278670992s) [seg_pos={0 0}, upload=496515]: Begin upload segment... Dec 29 05:47:06.407654203 (566h18m6.278737238s) [seg_pos={0 0}, upload=496515]: Done begin upload segment. Dec 29 05:47:06.407687316 (566h18m6.278770351s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407739155 (566h18m6.27882219s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407836391 (566h18m6.278919426s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408219260 (566h18m6.279302295s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408593622 (566h18m6.279676657s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409254480 (566h18m6.280337515s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409689258 (566h18m6.280772294s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410118516 (566h18m6.281201561s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410504250 (566h18m6.281587295s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.411968288 (566h18m6.283051333s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.413176318 (566h18m6.284259363s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.414893808 (566h18m6.285976853s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416341014 (566h18m6.28742405s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416677855 (566h18m6.28776089s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416696692 (566h18m6.287779737s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416776694 (566h18m6.287859729s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:08.105624686 (566h18m7.976707731s) [upload=496515]: Next returned nil segment Dec 29 05:47:08.105645005 (566h18m7.97672805s) [upload=496515]: Waiting for error group managing segments... Dec 29 05:47:08.387857626 (566h18m8.258940671s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398533746 (566h18m8.269616791s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398556088 (566h18m8.269639133s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.400797887 (566h18m8.271880932s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.403560297 (566h18m8.274643342s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.405882128 (566h18m8.276965173s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.408808900 (566h18m8.279891945s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.419006578 (566h18m8.290089624s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.421649471 (566h18m8.292732516s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.422123274 (566h18m8.293206319s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.423940313 (566h18m8.295023348s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.656954976 (566h18m8.528038011s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725071939 (566h18m8.596154985s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725097759 (566h18m8.596180804s) [seg_pos={0 0}, upload=496515]: Segment reached optimal threshold of 13 pieces. Dec 29 05:47:08.725351091 (566h18m8.596434136s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725361110 (566h18m8.596444165s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725504133 (566h18m8.596587208s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725514573 (566h18m8.596597618s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725713051 (566h18m8.596796096s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725722368 (566h18m8.596805413s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725743438 (566h18m8.596826483s) [seg_pos={0 0}, upload=496515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:47:08.725873205 (566h18m8.59695626s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:47:08.725881431 (566h18m8.596964476s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:47:09.207281964 (566h18m9.078365009s) [upload=496515]: Done uploading segments... Dec 29 05:47:09.207379881 (566h18m9.078462926s) [upload=496515]: Upload finished. err: <nil> Duration: 5.947907209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X..X.X...XX...X.XX...........XXXX....X.X.X..X.........] duration: 6.024023137s trace: https://jaeger.dev.storj.io/trace/7c4c54f5bda64ae7 FAIL (2025-12-29 05:54:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 05:47:06.139220037 (566h18m6.010303092s) [upload=496515]: Starting upload Dec 29 05:47:06.139302664 (566h18m6.010385709s) [upload=496515]: Done starting upload Dec 29 05:47:06.139471035 (566h18m6.01055409s) [upload=496515]: Uploading segments... Dec 29 05:47:06.139568380 (566h18m6.010651435s) [upload=496515]: Got next segment. Inline: false Dec 29 05:47:06.139620329 (566h18m6.010703364s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:47:06.139626592 (566h18m6.010709637s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:47:06.407587957 (566h18m6.278670992s) [seg_pos={0 0}, upload=496515]: Begin upload segment... Dec 29 05:47:06.407654203 (566h18m6.278737238s) [seg_pos={0 0}, upload=496515]: Done begin upload segment. Dec 29 05:47:06.407687316 (566h18m6.278770351s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407739155 (566h18m6.27882219s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407836391 (566h18m6.278919426s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408219260 (566h18m6.279302295s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408593622 (566h18m6.279676657s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409254480 (566h18m6.280337515s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409689258 (566h18m6.280772294s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410118516 (566h18m6.281201561s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410504250 (566h18m6.281587295s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.411968288 (566h18m6.283051333s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.413176318 (566h18m6.284259363s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.414893808 (566h18m6.285976853s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416341014 (566h18m6.28742405s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416677855 (566h18m6.28776089s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416696692 (566h18m6.287779737s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416776694 (566h18m6.287859729s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:08.105624686 (566h18m7.976707731s) [upload=496515]: Next returned nil segment Dec 29 05:47:08.105645005 (566h18m7.97672805s) [upload=496515]: Waiting for error group managing segments... Dec 29 05:47:08.387857626 (566h18m8.258940671s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398533746 (566h18m8.269616791s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398556088 (566h18m8.269639133s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.400797887 (566h18m8.271880932s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.403560297 (566h18m8.274643342s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.405882128 (566h18m8.276965173s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.408808900 (566h18m8.279891945s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.419006578 (566h18m8.290089624s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.421649471 (566h18m8.292732516s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.422123274 (566h18m8.293206319s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.423940313 (566h18m8.295023348s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.656954976 (566h18m8.528038011s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725071939 (566h18m8.596154985s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725097759 (566h18m8.596180804s) [seg_pos={0 0}, upload=496515]: Segment reached optimal threshold of 13 pieces. Dec 29 05:47:08.725351091 (566h18m8.596434136s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725361110 (566h18m8.596444165s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725504133 (566h18m8.596587208s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725514573 (566h18m8.596597618s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725713051 (566h18m8.596796096s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725722368 (566h18m8.596805413s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725743438 (566h18m8.596826483s) [seg_pos={0 0}, upload=496515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:47:08.725873205 (566h18m8.59695626s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:47:08.725881431 (566h18m8.596964476s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:47:09.207281964 (566h18m9.078365009s) [upload=496515]: Done uploading segments... Dec 29 05:47:09.207379881 (566h18m9.078462926s) [upload=496515]: Upload finished. err: <nil> Duration: 6.024052223s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.....X..X...X....................................X.X....] duration: 37.916090506s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:53:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 37.916119362s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.........X...X...........X.....X...X.....XX.X.....X......XX.] duration: 3.610870195s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 05:53:51 UTC) orbiter warning: context deadline exceeded Duration: 3.610894931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......X..X.X...XX...X.XX...........XXXX....X.X.X..X........] duration: 5.6700729s trace: https://jaeger.dev.storj.io/trace/7c4c54f5bda64ae7 FAIL (2025-12-29 05:53:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 05:47:06.139220037 (566h18m6.010303092s) [upload=496515]: Starting upload Dec 29 05:47:06.139302664 (566h18m6.010385709s) [upload=496515]: Done starting upload Dec 29 05:47:06.139471035 (566h18m6.01055409s) [upload=496515]: Uploading segments... Dec 29 05:47:06.139568380 (566h18m6.010651435s) [upload=496515]: Got next segment. Inline: false Dec 29 05:47:06.139620329 (566h18m6.010703364s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:47:06.139626592 (566h18m6.010709637s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:47:06.407587957 (566h18m6.278670992s) [seg_pos={0 0}, upload=496515]: Begin upload segment... Dec 29 05:47:06.407654203 (566h18m6.278737238s) [seg_pos={0 0}, upload=496515]: Done begin upload segment. Dec 29 05:47:06.407687316 (566h18m6.278770351s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407739155 (566h18m6.27882219s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407836391 (566h18m6.278919426s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408219260 (566h18m6.279302295s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408593622 (566h18m6.279676657s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409254480 (566h18m6.280337515s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409689258 (566h18m6.280772294s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410118516 (566h18m6.281201561s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410504250 (566h18m6.281587295s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.411968288 (566h18m6.283051333s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.413176318 (566h18m6.284259363s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.414893808 (566h18m6.285976853s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416341014 (566h18m6.28742405s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416677855 (566h18m6.28776089s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416696692 (566h18m6.287779737s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416776694 (566h18m6.287859729s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:08.105624686 (566h18m7.976707731s) [upload=496515]: Next returned nil segment Dec 29 05:47:08.105645005 (566h18m7.97672805s) [upload=496515]: Waiting for error group managing segments... Dec 29 05:47:08.387857626 (566h18m8.258940671s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398533746 (566h18m8.269616791s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398556088 (566h18m8.269639133s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.400797887 (566h18m8.271880932s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.403560297 (566h18m8.274643342s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.405882128 (566h18m8.276965173s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.408808900 (566h18m8.279891945s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.419006578 (566h18m8.290089624s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.421649471 (566h18m8.292732516s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.422123274 (566h18m8.293206319s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.423940313 (566h18m8.295023348s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.656954976 (566h18m8.528038011s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725071939 (566h18m8.596154985s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725097759 (566h18m8.596180804s) [seg_pos={0 0}, upload=496515]: Segment reached optimal threshold of 13 pieces. Dec 29 05:47:08.725351091 (566h18m8.596434136s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725361110 (566h18m8.596444165s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725504133 (566h18m8.596587208s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725514573 (566h18m8.596597618s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725713051 (566h18m8.596796096s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725722368 (566h18m8.596805413s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725743438 (566h18m8.596826483s) [seg_pos={0 0}, upload=496515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:47:08.725873205 (566h18m8.59695626s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:47:08.725881431 (566h18m8.596964476s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:47:09.207281964 (566h18m9.078365009s) [upload=496515]: Done uploading segments... Dec 29 05:47:09.207379881 (566h18m9.078462926s) [upload=496515]: Upload finished. err: <nil> Duration: 5.67009884s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X.........X...X...........X.....X...X.....XX.X.....X......XX] duration: 5.001491856s trace: https://jaeger.dev.storj.io/trace/4a5e01105e522140 FAIL (2025-12-29 05:53:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001513857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.....X..X...X....................................X.X...] duration: 47.574807347s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:53:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 47.574837485s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........X..X.X...XX...X.XX...........XXXX....X.X.X..X.......] duration: 5.754774926s trace: https://jaeger.dev.storj.io/trace/7c4c54f5bda64ae7 FAIL (2025-12-29 05:52:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 05:47:06.139220037 (566h18m6.010303092s) [upload=496515]: Starting upload Dec 29 05:47:06.139302664 (566h18m6.010385709s) [upload=496515]: Done starting upload Dec 29 05:47:06.139471035 (566h18m6.01055409s) [upload=496515]: Uploading segments... Dec 29 05:47:06.139568380 (566h18m6.010651435s) [upload=496515]: Got next segment. Inline: false Dec 29 05:47:06.139620329 (566h18m6.010703364s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:47:06.139626592 (566h18m6.010709637s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:47:06.407587957 (566h18m6.278670992s) [seg_pos={0 0}, upload=496515]: Begin upload segment... Dec 29 05:47:06.407654203 (566h18m6.278737238s) [seg_pos={0 0}, upload=496515]: Done begin upload segment. Dec 29 05:47:06.407687316 (566h18m6.278770351s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407739155 (566h18m6.27882219s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407836391 (566h18m6.278919426s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408219260 (566h18m6.279302295s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408593622 (566h18m6.279676657s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409254480 (566h18m6.280337515s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409689258 (566h18m6.280772294s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410118516 (566h18m6.281201561s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410504250 (566h18m6.281587295s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.411968288 (566h18m6.283051333s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.413176318 (566h18m6.284259363s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.414893808 (566h18m6.285976853s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416341014 (566h18m6.28742405s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416677855 (566h18m6.28776089s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416696692 (566h18m6.287779737s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416776694 (566h18m6.287859729s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:08.105624686 (566h18m7.976707731s) [upload=496515]: Next returned nil segment Dec 29 05:47:08.105645005 (566h18m7.97672805s) [upload=496515]: Waiting for error group managing segments... Dec 29 05:47:08.387857626 (566h18m8.258940671s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398533746 (566h18m8.269616791s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398556088 (566h18m8.269639133s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.400797887 (566h18m8.271880932s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.403560297 (566h18m8.274643342s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.405882128 (566h18m8.276965173s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.408808900 (566h18m8.279891945s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.419006578 (566h18m8.290089624s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.421649471 (566h18m8.292732516s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.422123274 (566h18m8.293206319s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.423940313 (566h18m8.295023348s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.656954976 (566h18m8.528038011s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725071939 (566h18m8.596154985s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725097759 (566h18m8.596180804s) [seg_pos={0 0}, upload=496515]: Segment reached optimal threshold of 13 pieces. Dec 29 05:47:08.725351091 (566h18m8.596434136s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725361110 (566h18m8.596444165s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725504133 (566h18m8.596587208s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725514573 (566h18m8.596597618s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725713051 (566h18m8.596796096s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725722368 (566h18m8.596805413s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725743438 (566h18m8.596826483s) [seg_pos={0 0}, upload=496515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:47:08.725873205 (566h18m8.59695626s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:47:08.725881431 (566h18m8.596964476s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:47:09.207281964 (566h18m9.078365009s) [upload=496515]: Done uploading segments... Dec 29 05:47:09.207379881 (566h18m9.078462926s) [upload=496515]: Upload finished. err: <nil> Duration: 5.754800636s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X.........X...X...........X.....X...X.....XX.X.....X......X] duration: 5.001470195s trace: https://jaeger.dev.storj.io/trace/663be07c0e2ef656 FAIL (2025-12-29 05:52:37 UTC) orbiter warning: context deadline exceeded Duration: 5.001493379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..X.........X...X...........X.....X...X.....XX.X.....X......] duration: 3.543616982s trace: https://jaeger.dev.storj.io/trace/33ddacb15db8f406 FAIL (2025-12-29 05:52:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.543644545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.....X..X...X....................................X.X..] duration: 30.634917942s trace: https://jaeger.dev.storj.io/trace/48b928ceba386a4f FAIL (2025-12-29 05:51:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867606615 (566h20m14.73868965s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.867674103 (566h20m14.738757138s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868507330 (566h20m14.739590365s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.868854311 (566h20m14.739937346s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869152479 (566h20m14.740235514s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869424877 (566h20m14.740507912s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869683851 (566h20m14.740766886s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.869908257 (566h20m14.740991293s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870201416 (566h20m14.741284451s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870481830 (566h20m14.741564865s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.870844821 (566h20m14.741927856s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.872942405 (566h20m14.74402544s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873321687 (566h20m14.744404732s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.873646426 (566h20m14.744729461s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:14.874099769 (566h20m14.745182804s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Uploading piece... Dec 29 05:49:16.187628230 (566h20m16.058711265s) [upload=496549]: Next returned nil segment Dec 29 05:49:16.187640323 (566h20m16.058723358s) [upload=496549]: Waiting for error group managing segments... Dec 29 05:49:16.720282267 (566h20m16.591365302s) [noise=true , address=145.79.227.14:4859 , piece_id=NHLMLCDC3BGPT37ZXPSQET7QIXSTKM65KIHWKAJBCCBXITJGW4KA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.726711199 (566h20m16.597794244s) [noise=true , address=145.79.227.4:4213 , piece_id=OG2VNO7MIH5VTVZ653SCXWXBNP6NQIKUB33BFHCLCLIFXGDDTSBQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.733680419 (566h20m16.604763484s) [noise=true , address=145.79.227.13:4835 , piece_id=ZSBGWPV7MYD5LCJOPY4YI66OF7C5VFGE2FIJSLWEXC2B3CDT3HFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.736676564 (566h20m16.60775962s) [noise=true , address=145.79.227.10:4425 , piece_id=YAPA6SNRUCPGVB7LB6WLYULAJCZYXQCYRHCFII6MTEZNP5G4EH2A, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.749841152 (566h20m16.620924187s) [noise=true , address=145.79.227.3:4162 , piece_id=3ISSHPSPYFKINMNU5NH5GUHPBWZ2IN5M5DDI6YMVJG562ZJQAHGA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.961927233 (566h20m16.833010268s) [noise=true , address=45.91.116.10:4097 , piece_id=HHXUFZWURWVFAHH5J2OLJIIB4ZMVZ4GM4RGEQJSLDQ6RVUBX6KWQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:16.999732149 (566h20m16.870815184s) [noise=true , address=45.91.116.13:4178 , piece_id=AXOKOYJKQRPGFMO54YSQOJGVTTDQQ7R57A4CTH5FZCXRYHOYCXXQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.014718495 (566h20m16.88580153s) [noise=true , address=77.90.31.15:4710 , piece_id=ZYNPDO6Q7XPUCSO6LDU4BRPZHW6OAZSPK2ZEUYUHZQSFHMLYLUHQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.029593590 (566h20m16.900676625s) [noise=true , address=77.90.31.14:4676 , piece_id=CRX7GO25CEWCA63DHQ4IOGBYACZU7HB7UOLB5A4UCU2C237EKM2Q, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.477941807 (566h20m17.349024852s) [noise=true , address=193.149.164.143:4704 , piece_id=XR2CQ5D2AZT7PEOMYODF4K437CETWV3B52OKQTW3P2PBBNPBVAFA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.629345851 (566h20m17.500428887s) [noise=true , address=145.79.227.49:4365 , piece_id=QFR7NNUMHKWNC4YKC7KBWM53O2FOIY6X4FOQ4BCAIJICUCYMHPRA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.636380477 (566h20m17.507463512s) [noise=true , address=145.79.227.30:4768 , piece_id=PIJF4XHQ72LUOFLJQXUDPZINUAX2AIDD3ATFJ3WZXP6U4MOXLTJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.665059636 (566h20m17.536142671s) [noise=true , address=145.79.227.40:4058 , piece_id=MI4UJ5CEWEOX2BLLWWOQTTX73HMGSTYTZZNGWLIVNIX4QQSVB6ZA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684608810 (566h20m17.555691845s) [noise=true , address=145.79.227.33:4903 , piece_id=F7LVHJ54LX2EXKGD5EUQTF2URZTCXNME4VRF7SUNNHKNV2L2SLIQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: <nil> Dec 29 05:49:17.684633297 (566h20m17.555716332s) [seg_pos={0 0}, upload=496549]: Segment reached optimal threshold of 14 pieces. Dec 29 05:49:17.684852214 (566h20m17.555935249s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.684859918 (566h20m17.555942953s) [noise=true , address=145.79.227.44:4209 , piece_id=LZQXQGNCRAHMHSRQFCAN36AECFSXO7H4S5LIUADUGPLC7YPFTOXA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.684977793 (566h20m17.556060828s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->64.49.13.169:4705: use of closed network connection Dec 29 05:49:17.684984075 (566h20m17.55606711s) [noise=true , address=64.49.13.169:4705 , piece_id=ARSV3OU7XWH2PGXA63ENIYJFEXEMQW5DDKS3T7O4SKJ5B7TBHLKQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685004744 (566h20m17.556087789s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685025784 (566h20m17.556108819s) [noise=true , address=77.90.31.19:4855 , piece_id=QQLTTKPUM4TCIMMVI47VCUFSHKIX3V7UHN4CXQ4QL5ICIMRTCOJA, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685157886 (566h20m17.556240931s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG): protocol: expected piece hash; context canceled; EOF Dec 29 05:49:17.685167284 (566h20m17.556250319s) [noise=true , address=145.79.227.43:4188 , piece_id=J6RIFLCA6FW6UVP6X2QNTSIKYVNLDX5IBW24P2HMR5LDMTFXYKRQ, seg_pos={0 0}, upload=496549]: Long tail cancellation due to optimal threshold Dec 29 05:49:17.685274147 (566h20m17.556357182s) [seg_pos={0 0}, upload=496549]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:49:17.685349370 (566h20m17.556432415s) [upload=496549]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:49:17.685354159 (566h20m17.556437194s) [upload=496549]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:49:17.988224297 (566h20m17.859307332s) [upload=496549]: Done uploading segments... Dec 29 05:49:17.988247982 (566h20m17.859331017s) [upload=496549]: Upload finished. err: <nil> Duration: 30.634959101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...X.........X...X...........X.....X...X.....XX.X.....X.....] duration: 5.000580553s trace: https://jaeger.dev.storj.io/trace/33ddacb15db8f406 FAIL (2025-12-29 05:51:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000597835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.........X..X.X...XX...X.XX...........XXXX....X.X.X..X......] duration: 5.921931637s trace: https://jaeger.dev.storj.io/trace/7c4c54f5bda64ae7 FAIL (2025-12-29 05:51:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 05:47:06.139220037 (566h18m6.010303092s) [upload=496515]: Starting upload Dec 29 05:47:06.139302664 (566h18m6.010385709s) [upload=496515]: Done starting upload Dec 29 05:47:06.139471035 (566h18m6.01055409s) [upload=496515]: Uploading segments... Dec 29 05:47:06.139568380 (566h18m6.010651435s) [upload=496515]: Got next segment. Inline: false Dec 29 05:47:06.139620329 (566h18m6.010703364s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:47:06.139626592 (566h18m6.010709637s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:47:06.407587957 (566h18m6.278670992s) [seg_pos={0 0}, upload=496515]: Begin upload segment... Dec 29 05:47:06.407654203 (566h18m6.278737238s) [seg_pos={0 0}, upload=496515]: Done begin upload segment. Dec 29 05:47:06.407687316 (566h18m6.278770351s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407739155 (566h18m6.27882219s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.407836391 (566h18m6.278919426s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408219260 (566h18m6.279302295s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.408593622 (566h18m6.279676657s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409254480 (566h18m6.280337515s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.409689258 (566h18m6.280772294s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410118516 (566h18m6.281201561s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.410504250 (566h18m6.281587295s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.411968288 (566h18m6.283051333s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.413176318 (566h18m6.284259363s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.414893808 (566h18m6.285976853s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416341014 (566h18m6.28742405s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416677855 (566h18m6.28776089s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416696692 (566h18m6.287779737s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:06.416776694 (566h18m6.287859729s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Uploading piece... Dec 29 05:47:08.105624686 (566h18m7.976707731s) [upload=496515]: Next returned nil segment Dec 29 05:47:08.105645005 (566h18m7.97672805s) [upload=496515]: Waiting for error group managing segments... Dec 29 05:47:08.387857626 (566h18m8.258940671s) [noise=true , address=145.79.227.2:4115 , piece_id=CDCRR7L6ABYYFSYLH235VCDLC7ELWZOFWOOFI3G5XJOKRORZHZMA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398533746 (566h18m8.269616791s) [noise=true , address=145.79.227.15:4876 , piece_id=7DKVH63ELTIP6EQPQLD337VE3CTSIOWHS7YLJSUZGN5DJTQPXCKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.398556088 (566h18m8.269639133s) [noise=true , address=145.79.227.5:4223 , piece_id=6QI7MFVAGOJQM6OYA4WFLNWUZKBWYPAUFI755GXEPCOB2QIZYURQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.400797887 (566h18m8.271880932s) [noise=true , address=145.79.227.20:4438 , piece_id=PQXMC2OHMJ2F7RSWVD76YAJHNA7KJHG5S4PEAUNS6UITLAVY7AGQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.403560297 (566h18m8.274643342s) [noise=true , address=145.79.227.37:4566 , piece_id=NFBPDVSOLKAF4EPWHCPGML4DH7UHOIN75CMHLPDYX2YODFNO3PFA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.405882128 (566h18m8.276965173s) [noise=true , address=145.79.227.46:4603 , piece_id=CLNN26LD5ECOGQTQKHAS7JYJTSHNB3RSMJX43MGPKWLIS7AIACKA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.408808900 (566h18m8.279891945s) [noise=true , address=145.79.227.26:4655 , piece_id=7C6VSBT7XAEIX2YW2Q67BLCRHRFMFZD7GFFZRT6DSFCAF46BOTMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.419006578 (566h18m8.290089624s) [noise=true , address=145.79.227.39:4013 , piece_id=YUKH2WSZ5SOHF6WMUOEWOBBVNRUJFFU7JCSF6DTZ32OE25PZSHMQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.421649471 (566h18m8.292732516s) [noise=true , address=145.79.227.16:4912 , piece_id=VYH2VOMGI2O77IYISS54N6MHZSC3SADAG63VWMQLAYBBTGRZE5WA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.422123274 (566h18m8.293206319s) [noise=true , address=145.79.227.52:4447 , piece_id=OU2S4OI4FDZ4TNMRKMBXVBLUPFEHWNMI5ZAWERIAKMDHJLOJCWYA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.423940313 (566h18m8.295023348s) [noise=true , address=145.79.227.4:4186 , piece_id=4SXH4WW6RINPDV5MM4VSQNSTCDGIHQRB6KJWOATCZWRGQFIORO4A, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.656954976 (566h18m8.528038011s) [noise=true , address=45.91.116.9:4264 , piece_id=QXYD7ZXJ26HFFDGMHIYIZR6JT6FTS5GQKGCILNOCT6ALMEHYEFVA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725071939 (566h18m8.596154985s) [noise=true , address=45.91.116.14:4191 , piece_id=BAJWGI4GDO7F7BONV7P45AXVYF6XRBTLSSXFXESHJ3HV4WBYKBXA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: <nil> Dec 29 05:47:08.725097759 (566h18m8.596180804s) [seg_pos={0 0}, upload=496515]: Segment reached optimal threshold of 13 pieces. Dec 29 05:47:08.725351091 (566h18m8.596434136s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725361110 (566h18m8.596444165s) [noise=true , address=77.90.31.13:4627 , piece_id=XAV3TKPLNEU3DCFFU6VHUHZQP2DX74JYZAFVFW6I7SQ7EHCP3NUA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725504133 (566h18m8.596587208s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725514573 (566h18m8.596597618s) [noise=true , address=77.90.31.14:4663 , piece_id=A4OXUAS6R6I2XBHZBWUPGCCXEIVVFYXRKPXMHVMR6N4CZJGJD7UA, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725713051 (566h18m8.596796096s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 05:47:08.725722368 (566h18m8.596805413s) [noise=true , address=45.91.116.10:4084 , piece_id=LBEOIE34K34C7BVDSNV7QJ3GVJZLMJOPZEZPVZCQDXPEAU4WXJIQ, seg_pos={0 0}, upload=496515]: Long tail cancellation due to optimal threshold Dec 29 05:47:08.725743438 (566h18m8.596826483s) [seg_pos={0 0}, upload=496515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:47:08.725873205 (566h18m8.59695626s) [upload=496515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:47:08.725881431 (566h18m8.596964476s) [upload=496515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:47:09.207281964 (566h18m9.078365009s) [upload=496515]: Done uploading segments... Dec 29 05:47:09.207379881 (566h18m9.078462926s) [upload=496515]: Upload finished. err: <nil> Duration: 5.921958949s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)