Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 349013 reports (Page 185 of 6981)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 11.825415441s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 15:00:35 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.825442101s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX.........] duration: 24.370552386s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 15:00:15 UTC) orbiter warning: context deadline exceeded Duration: 24.370585289s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 27.660904923s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:59:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 27.660928278s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................X........X.........X......X.......X...X] duration: 1m0.002613848s trace: https://jaeger.dev.storj.io/trace/24cdf967807c73aa FAIL (2025-12-29 14:59:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, Piece ID: 3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A) context deadline exceeded; piecestore: (Node ID: 12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD, Piece ID: WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ) context deadline exceeded; piecestore: (Node ID: 124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, Piece ID: D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q) context deadline exceeded ======LOGS====== 575h29m57.402023822s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531386685 (575h29m57.40246972s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.531810814 (575h29m57.402893859s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532173644 (575h29m57.403256679s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532505637 (575h29m57.403588682s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532521497 (575h29m57.403604542s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.532896561 (575h29m57.403979596s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533085691 (575h29m57.404168726s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533316711 (575h29m57.404399746s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.533620860 (575h29m57.404703906s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534046441 (575h29m57.405129486s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534231012 (575h29m57.405314047s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.534677893 (575h29m57.405760928s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535153379 (575h29m57.406236414s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.535639023 (575h29m57.406722058s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:57.536090052 (575h29m57.407173087s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Uploading piece... Dec 29 14:58:58.829579348 (575h29m58.700662383s) [upload=504500]: Next returned nil segment Dec 29 14:58:58.829609104 (575h29m58.700692149s) [upload=504500]: Waiting for error group managing segments... Dec 29 14:58:59.382931381 (575h29m59.254014416s) [noise=true , address=145.79.227.7:4304 , piece_id=SBP3BSDFDIWNFMEMFDDECFR27TA327GJXXFR64SJBCCQEKM4GT6A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.588655992 (575h29m59.459739027s) [noise=true , address=145.79.227.24:4613 , piece_id=DMJJPJ5ZZMTIYPRRC7HC2GBFDOHCPQBGHPLTT5DPYTO7KWZ6XXDQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.653654779 (575h29m59.524737814s) [noise=true , address=77.90.31.12:4599 , piece_id=D5IXRYSO6AHXIYMT2FC6PKBTMLEG6WSN7HNPCXDSGRX2AMD67U2Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:58:59.734453232 (575h29m59.605536277s) [noise=true , address=77.90.31.18:4819 , piece_id=3SZJVSYZR76V5NJGRXWUPX6CZ4ZK4Z5HW6YIPF3GAYFJZP2OWZ3A, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.167103872 (575h30m0.038186917s) [noise=true , address=64.49.13.26:4946 , piece_id=NVNWMHYLPL4E2ESGMCE2IWZTD3E2G726QEA7E3DMQRE6RIEXK4WQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.177447108 (575h30m0.048530173s) [noise=true , address=193.149.164.143:4701 , piece_id=EV2DMT7VGRI2WUO3ME6E23FRFFSDJLFJBNYTIHN2D376YIMVE7JA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.328386488 (575h30m0.199469533s) [noise=true , address=145.79.227.35:4493 , piece_id=LQAVTRENXCYGQYQZ32BMJTXRNRJMVVON4T6WTLEBM3QJETLZ2NTQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.361243262 (575h30m0.232326307s) [noise=true , address=45.91.116.5:4323 , piece_id=WGJ6BJ5TETWNIIJRKEGRAV3GNFNKBCEKKILTEGFUM55FHW3MFFYQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366336290 (575h30m0.237419325s) [noise=true , address=145.79.227.30:4769 , piece_id=OU7UEWD2LRU3HXGNNH4HOTSHUOSOYGFGB6BF5RNYW6CMOBY2MIAA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.366810864 (575h30m0.237893899s) [noise=true , address=145.79.227.43:4165 , piece_id=VNZFJVMPHS3II2EZRHSI4DQX5GNHA2GDU3IKB4D6I5EBQ4ZZEW5Q, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.382979812 (575h30m0.254062847s) [noise=true , address=145.79.227.34:4939 , piece_id=MTXSTBVZ3B4C2C6JAIKOMSQ2UBESHXR3IVU3ZHQBVLGGAMPYFIMQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.398425453 (575h30m0.269508488s) [noise=true , address=145.79.227.44:4228 , piece_id=FTUOW4SETJW53ZGZCDYFONEEZNLPPVCOEUXOZOID5H4JLHW72CBA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.459971035 (575h30m0.33105407s) [noise=true , address=145.79.227.29:4971 , piece_id=S4CHAEBLAAW5UZ55GBRUASI7EGCRED3XFQO3QEBYJULKGA7DPFSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463847726 (575h30m0.334930772s) [noise=true , address=77.90.31.15:4726 , piece_id=SR75W7NXQ3LEYAHFXODUZFOPYVLPVGE44MS3OWO64QB72YYXZGQA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: <nil> Dec 29 14:59:00.463867004 (575h30m0.334950039s) [seg_pos={0 0}, upload=504500]: Segment reached optimal threshold of 14 pieces. Dec 29 14:59:00.464178006 (575h30m0.335261051s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R6A6f6LTL3VDDg5gbhvQE55bRLJPDfAjViixeNX4KjVJpYFDE): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464187323 (575h30m0.335270358s) [noise=true , address=5.231.73.3:4017 , piece_id=DAOMTOT3AX4IGDJHULGYX2R2CPLE7J4S7TCUND2LGGLYUJ7FFUCQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464300679 (575h30m0.335383724s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464318744 (575h30m0.335401839s) [noise=true , address=145.79.227.33:4898 , piece_id=ZIVIAZHLGGMFGUPCSEBHPUS35WLDWUGUSGB6RPD7PTXMVQCGWVUQ, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464390671 (575h30m0.335473726s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF Dec 29 14:59:00.464401150 (575h30m0.335484205s) [noise=true , address=145.79.227.40:4070 , piece_id=YZHWBHD24QLHMUEWGPJIBK7XYTX5JOH3473AIDGBXVYSQZ5RNNFA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464498576 (575h30m0.335581621s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51812->64.49.13.169:4706: use of closed network connection Dec 29 14:59:00.464511450 (575h30m0.335594485s) [noise=true , address=64.49.13.169:4706 , piece_id=47LNMLBT6CDSALPEB7YHEQG2NWGYECZ66OGMPZTBWR4L5FVUYCSA, seg_pos={0 0}, upload=504500]: Long tail cancellation due to optimal threshold Dec 29 14:59:00.464535005 (575h30m0.33561804s) [seg_pos={0 0}, upload=504500]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:59:00.464667398 (575h30m0.335750443s) [upload=504500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:59:00.464674982 (575h30m0.335758017s) [upload=504500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:59:00.771413457 (575h30m0.642496502s) [upload=504500]: Done uploading segments... Dec 29 14:59:00.771439016 (575h30m0.642522051s) [upload=504500]: Upload finished. err: <nil> Duration: 1m0.002623727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration: 5.001288875s trace: https://jaeger.dev.storj.io/trace/2200bb03b1d7bba7 FAIL (2025-12-29 14:59:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001312219s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.....] duration: 7.821117076s trace: https://jaeger.dev.storj.io/trace/6eb6a4a5ee96c354 FAIL (2025-12-29 14:59:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 29 14:55:39.029548108 (575h26m38.900631153s) [upload=504459]: Got next segment. Inline: false Dec 29 14:55:39.029620336 (575h26m38.900703381s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:55:39.029628863 (575h26m38.900711898s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:55:39.274058943 (575h26m39.145141978s) [seg_pos={0 0}, upload=504459]: Begin upload segment... Dec 29 14:55:39.274103809 (575h26m39.145186844s) [seg_pos={0 0}, upload=504459]: Done begin upload segment. Dec 29 14:55:39.274118707 (575h26m39.145201752s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274171288 (575h26m39.145254323s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274211514 (575h26m39.145294549s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275556354 (575h26m39.146639389s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275649141 (575h26m39.146732176s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276045065 (575h26m39.14712811s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276399980 (575h26m39.147483016s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276513698 (575h26m39.147596733s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276936142 (575h26m39.148019187s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.277332496 (575h26m39.148415531s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.286003968 (575h26m39.157087003s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287295517 (575h26m39.158378553s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287744863 (575h26m39.158827898s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.288054283 (575h26m39.159137328s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.289285316 (575h26m39.160368361s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.290576334 (575h26m39.16165938s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:40.994006767 (575h26m40.865089802s) [upload=504459]: Next returned nil segment Dec 29 14:55:40.994030793 (575h26m40.865113828s) [upload=504459]: Waiting for error group managing segments... Dec 29 14:55:41.267833074 (575h26m41.138916119s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.281060259 (575h26m41.152143305s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.286387874 (575h26m41.157470919s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.287299309 (575h26m41.158382344s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.301675123 (575h26m41.172758169s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.302980108 (575h26m41.174063143s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.315675352 (575h26m41.186758387s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.325229314 (575h26m41.196312349s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.330815401 (575h26m41.201898436s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.341338648 (575h26m41.212421683s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.347929017 (575h26m41.219012062s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.623535403 (575h26m41.494618438s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628522058 (575h26m41.499605093s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628547236 (575h26m41.499630272s) [seg_pos={0 0}, upload=504459]: Segment reached optimal threshold of 13 pieces. Dec 29 14:55:41.628757737 (575h26m41.499840782s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:41.628767966 (575h26m41.499851001s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.628945805 (575h26m41.50002885s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection Dec 29 14:55:41.628954733 (575h26m41.500037778s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629177076 (575h26m41.500260111s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection Dec 29 14:55:41.629199218 (575h26m41.500282273s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629231810 (575h26m41.500314855s) [seg_pos={0 0}, upload=504459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:55:41.629342471 (575h26m41.500425506s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:41.629350496 (575h26m41.500433531s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:41.923681265 (575h26m41.79476431s) [upload=504459]: Done uploading segments... Dec 29 14:55:41.923704099 (575h26m41.794787134s) [upload=504459]: Upload finished. err: <nil> Duration: 7.8211457s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.............................XXX...........XX.X.............] duration: 28.15765317s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:59:31 UTC) orbiter warning: context deadline exceeded Duration: 28.157670232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX....] duration: 5.969443514s trace: https://jaeger.dev.storj.io/trace/6eb6a4a5ee96c354 FAIL (2025-12-29 14:59:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 29 14:55:39.029548108 (575h26m38.900631153s) [upload=504459]: Got next segment. Inline: false Dec 29 14:55:39.029620336 (575h26m38.900703381s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:55:39.029628863 (575h26m38.900711898s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:55:39.274058943 (575h26m39.145141978s) [seg_pos={0 0}, upload=504459]: Begin upload segment... Dec 29 14:55:39.274103809 (575h26m39.145186844s) [seg_pos={0 0}, upload=504459]: Done begin upload segment. Dec 29 14:55:39.274118707 (575h26m39.145201752s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274171288 (575h26m39.145254323s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274211514 (575h26m39.145294549s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275556354 (575h26m39.146639389s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275649141 (575h26m39.146732176s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276045065 (575h26m39.14712811s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276399980 (575h26m39.147483016s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276513698 (575h26m39.147596733s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276936142 (575h26m39.148019187s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.277332496 (575h26m39.148415531s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.286003968 (575h26m39.157087003s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287295517 (575h26m39.158378553s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287744863 (575h26m39.158827898s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.288054283 (575h26m39.159137328s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.289285316 (575h26m39.160368361s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.290576334 (575h26m39.16165938s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:40.994006767 (575h26m40.865089802s) [upload=504459]: Next returned nil segment Dec 29 14:55:40.994030793 (575h26m40.865113828s) [upload=504459]: Waiting for error group managing segments... Dec 29 14:55:41.267833074 (575h26m41.138916119s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.281060259 (575h26m41.152143305s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.286387874 (575h26m41.157470919s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.287299309 (575h26m41.158382344s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.301675123 (575h26m41.172758169s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.302980108 (575h26m41.174063143s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.315675352 (575h26m41.186758387s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.325229314 (575h26m41.196312349s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.330815401 (575h26m41.201898436s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.341338648 (575h26m41.212421683s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.347929017 (575h26m41.219012062s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.623535403 (575h26m41.494618438s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628522058 (575h26m41.499605093s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628547236 (575h26m41.499630272s) [seg_pos={0 0}, upload=504459]: Segment reached optimal threshold of 13 pieces. Dec 29 14:55:41.628757737 (575h26m41.499840782s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:41.628767966 (575h26m41.499851001s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.628945805 (575h26m41.50002885s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection Dec 29 14:55:41.628954733 (575h26m41.500037778s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629177076 (575h26m41.500260111s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection Dec 29 14:55:41.629199218 (575h26m41.500282273s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629231810 (575h26m41.500314855s) [seg_pos={0 0}, upload=504459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:55:41.629342471 (575h26m41.500425506s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:41.629350496 (575h26m41.500433531s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:41.923681265 (575h26m41.79476431s) [upload=504459]: Done uploading segments... Dec 29 14:55:41.923704099 (575h26m41.794787134s) [upload=504459]: Upload finished. err: <nil> Duration: 5.969469444s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..............................XXX...........XX.X............] duration: 29.762936568s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:59:03 UTC) orbiter warning: context deadline exceeded Duration: 29.763098135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX........] duration: 16.898307171s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:59:02 UTC) orbiter warning: context deadline exceeded Duration: 16.898318494s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X........X.........X......X.......X...] duration: 49.581871703s trace: https://jaeger.dev.storj.io/trace/4d59fb908fcd5e32 FAIL (2025-12-29 14:58:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.201966488 (575h26m33.073049533s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.202316224 (575h26m33.073399259s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203100067 (575h26m33.074183112s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203517372 (575h26m33.074600417s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204138164 (575h26m33.075221209s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204587550 (575h26m33.075670595s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205402070 (575h26m33.076485115s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205695298 (575h26m33.076778333s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.206491314 (575h26m33.077574349s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207058744 (575h26m33.078141789s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207697070 (575h26m33.078780105s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208500180 (575h26m33.079583215s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208704328 (575h26m33.079787364s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.209759517 (575h26m33.080842552s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.210896212 (575h26m33.081979257s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:34.572355890 (575h26m34.443438925s) [upload=504454]: Next returned nil segment Dec 29 14:55:34.572368655 (575h26m34.44345169s) [upload=504454]: Waiting for error group managing segments... Dec 29 14:55:35.103639091 (575h26m34.974722126s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.255917912 (575h26m35.127000967s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.348450017 (575h26m35.219533062s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.519579483 (575h26m35.390662528s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.624938442 (575h26m35.496021487s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.702059762 (575h26m35.573142797s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.875749101 (575h26m35.746832156s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.978308259 (575h26m35.849391294s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.980555457 (575h26m35.851638492s) [noise=true , address=145.79.227.40:4057 , piece_id=YD6LHBJEUJCGRDYDYPSKYMYAXPDI7YXEFEIIX65R4X6I3FIGQBHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.006221009 (575h26m35.877304044s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.015400719 (575h26m35.886483754s) [noise=true , address=145.79.227.30:4781 , piece_id=VGRLYZCMCLDD6YTMENV3FDPUE5T3XZMY37EPZXCI3ZRPRGPESDZA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.034554079 (575h26m35.905637114s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.072374315 (575h26m35.94345735s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127854296 (575h26m35.998937331s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127872321 (575h26m35.998955356s) [seg_pos={0 0}, upload=504454]: Segment reached optimal threshold of 14 pieces. Dec 29 14:55:36.128098612 (575h26m35.999181647s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e7rvSxVxpjoe6zZeXYiVCyDs8sqB4DR1oWN7rGByAdUT9vjp): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.128105936 (575h26m35.999188971s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.128400156 (575h26m35.999483191s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection Dec 29 14:55:36.128406738 (575h26m35.999489773s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129158100 (575h26m36.000241135s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129165283 (575h26m36.000248318s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129232621 (575h26m36.000315656s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129239735 (575h26m36.00032277s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129261206 (575h26m36.000344241s) [seg_pos={0 0}, upload=504454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:55:36.129337591 (575h26m36.000420626s) [upload=504454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:36.129342060 (575h26m36.000425095s) [upload=504454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:36.429197408 (575h26m36.300280453s) [upload=504454]: Done uploading segments... Dec 29 14:55:36.429249858 (575h26m36.300332904s) [upload=504454]: Upload finished. err: <nil> Duration: 49.581894195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 13.653840166s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:58:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 13.653867937s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX.......] duration: 18.944169068s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:58:45 UTC) orbiter warning: context deadline exceeded Duration: 18.94419686s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration: 5.00051666s trace: https://jaeger.dev.storj.io/trace/40732a9f940e0851 FAIL (2025-12-29 14:58:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000540136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXXXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX...] duration: 5.80207739s trace: https://jaeger.dev.storj.io/trace/6eb6a4a5ee96c354 FAIL (2025-12-29 14:58:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 29 14:55:39.029548108 (575h26m38.900631153s) [upload=504459]: Got next segment. Inline: false Dec 29 14:55:39.029620336 (575h26m38.900703381s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:55:39.029628863 (575h26m38.900711898s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:55:39.274058943 (575h26m39.145141978s) [seg_pos={0 0}, upload=504459]: Begin upload segment... Dec 29 14:55:39.274103809 (575h26m39.145186844s) [seg_pos={0 0}, upload=504459]: Done begin upload segment. Dec 29 14:55:39.274118707 (575h26m39.145201752s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274171288 (575h26m39.145254323s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274211514 (575h26m39.145294549s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275556354 (575h26m39.146639389s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275649141 (575h26m39.146732176s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276045065 (575h26m39.14712811s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276399980 (575h26m39.147483016s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276513698 (575h26m39.147596733s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276936142 (575h26m39.148019187s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.277332496 (575h26m39.148415531s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.286003968 (575h26m39.157087003s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287295517 (575h26m39.158378553s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287744863 (575h26m39.158827898s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.288054283 (575h26m39.159137328s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.289285316 (575h26m39.160368361s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.290576334 (575h26m39.16165938s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:40.994006767 (575h26m40.865089802s) [upload=504459]: Next returned nil segment Dec 29 14:55:40.994030793 (575h26m40.865113828s) [upload=504459]: Waiting for error group managing segments... Dec 29 14:55:41.267833074 (575h26m41.138916119s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.281060259 (575h26m41.152143305s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.286387874 (575h26m41.157470919s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.287299309 (575h26m41.158382344s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.301675123 (575h26m41.172758169s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.302980108 (575h26m41.174063143s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.315675352 (575h26m41.186758387s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.325229314 (575h26m41.196312349s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.330815401 (575h26m41.201898436s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.341338648 (575h26m41.212421683s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.347929017 (575h26m41.219012062s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.623535403 (575h26m41.494618438s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628522058 (575h26m41.499605093s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628547236 (575h26m41.499630272s) [seg_pos={0 0}, upload=504459]: Segment reached optimal threshold of 13 pieces. Dec 29 14:55:41.628757737 (575h26m41.499840782s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:41.628767966 (575h26m41.499851001s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.628945805 (575h26m41.50002885s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection Dec 29 14:55:41.628954733 (575h26m41.500037778s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629177076 (575h26m41.500260111s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection Dec 29 14:55:41.629199218 (575h26m41.500282273s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629231810 (575h26m41.500314855s) [seg_pos={0 0}, upload=504459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:55:41.629342471 (575h26m41.500425506s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:41.629350496 (575h26m41.500433531s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:41.923681265 (575h26m41.79476431s) [upload=504459]: Done uploading segments... Dec 29 14:55:41.923704099 (575h26m41.794787134s) [upload=504459]: Upload finished. err: <nil> Duration: 5.802100252s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX......] duration: 23.183526034s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:58:13 UTC) orbiter warning: context deadline exceeded Duration: 23.183589014s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration: 5.002531239s trace: https://jaeger.dev.storj.io/trace/725eba0b4c5c0947 FAIL (2025-12-29 14:58:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002557781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X........X.........X......X.......X..] duration: 37.642485232s trace: https://jaeger.dev.storj.io/trace/4d59fb908fcd5e32 FAIL (2025-12-29 14:58:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.201966488 (575h26m33.073049533s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.202316224 (575h26m33.073399259s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203100067 (575h26m33.074183112s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203517372 (575h26m33.074600417s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204138164 (575h26m33.075221209s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204587550 (575h26m33.075670595s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205402070 (575h26m33.076485115s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205695298 (575h26m33.076778333s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.206491314 (575h26m33.077574349s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207058744 (575h26m33.078141789s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207697070 (575h26m33.078780105s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208500180 (575h26m33.079583215s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208704328 (575h26m33.079787364s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.209759517 (575h26m33.080842552s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.210896212 (575h26m33.081979257s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:34.572355890 (575h26m34.443438925s) [upload=504454]: Next returned nil segment Dec 29 14:55:34.572368655 (575h26m34.44345169s) [upload=504454]: Waiting for error group managing segments... Dec 29 14:55:35.103639091 (575h26m34.974722126s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.255917912 (575h26m35.127000967s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.348450017 (575h26m35.219533062s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.519579483 (575h26m35.390662528s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.624938442 (575h26m35.496021487s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.702059762 (575h26m35.573142797s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.875749101 (575h26m35.746832156s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.978308259 (575h26m35.849391294s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.980555457 (575h26m35.851638492s) [noise=true , address=145.79.227.40:4057 , piece_id=YD6LHBJEUJCGRDYDYPSKYMYAXPDI7YXEFEIIX65R4X6I3FIGQBHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.006221009 (575h26m35.877304044s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.015400719 (575h26m35.886483754s) [noise=true , address=145.79.227.30:4781 , piece_id=VGRLYZCMCLDD6YTMENV3FDPUE5T3XZMY37EPZXCI3ZRPRGPESDZA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.034554079 (575h26m35.905637114s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.072374315 (575h26m35.94345735s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127854296 (575h26m35.998937331s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127872321 (575h26m35.998955356s) [seg_pos={0 0}, upload=504454]: Segment reached optimal threshold of 14 pieces. Dec 29 14:55:36.128098612 (575h26m35.999181647s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e7rvSxVxpjoe6zZeXYiVCyDs8sqB4DR1oWN7rGByAdUT9vjp): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.128105936 (575h26m35.999188971s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.128400156 (575h26m35.999483191s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection Dec 29 14:55:36.128406738 (575h26m35.999489773s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129158100 (575h26m36.000241135s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129165283 (575h26m36.000248318s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129232621 (575h26m36.000315656s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129239735 (575h26m36.00032277s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129261206 (575h26m36.000344241s) [seg_pos={0 0}, upload=504454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:55:36.129337591 (575h26m36.000420626s) [upload=504454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:36.129342060 (575h26m36.000425095s) [upload=504454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:36.429197408 (575h26m36.300280453s) [upload=504454]: Done uploading segments... Dec 29 14:55:36.429249858 (575h26m36.300332904s) [upload=504454]: Upload finished. err: <nil> Duration: 37.642506252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.487342543s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:57:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.487363812s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X..............................XXX...........XX.X...........] duration: 25.291424961s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:57:51 UTC) orbiter warning: context deadline exceeded Duration: 25.291474815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX.....] duration: 22.420518139s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:57:50 UTC) orbiter warning: context deadline exceeded Duration: 22.420565109s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XXXXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX..] duration: 6.077788297s trace: https://jaeger.dev.storj.io/trace/6eb6a4a5ee96c354 FAIL (2025-12-29 14:57:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 29 14:55:39.029548108 (575h26m38.900631153s) [upload=504459]: Got next segment. Inline: false Dec 29 14:55:39.029620336 (575h26m38.900703381s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:55:39.029628863 (575h26m38.900711898s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:55:39.274058943 (575h26m39.145141978s) [seg_pos={0 0}, upload=504459]: Begin upload segment... Dec 29 14:55:39.274103809 (575h26m39.145186844s) [seg_pos={0 0}, upload=504459]: Done begin upload segment. Dec 29 14:55:39.274118707 (575h26m39.145201752s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274171288 (575h26m39.145254323s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274211514 (575h26m39.145294549s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275556354 (575h26m39.146639389s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275649141 (575h26m39.146732176s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276045065 (575h26m39.14712811s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276399980 (575h26m39.147483016s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276513698 (575h26m39.147596733s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276936142 (575h26m39.148019187s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.277332496 (575h26m39.148415531s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.286003968 (575h26m39.157087003s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287295517 (575h26m39.158378553s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287744863 (575h26m39.158827898s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.288054283 (575h26m39.159137328s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.289285316 (575h26m39.160368361s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.290576334 (575h26m39.16165938s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:40.994006767 (575h26m40.865089802s) [upload=504459]: Next returned nil segment Dec 29 14:55:40.994030793 (575h26m40.865113828s) [upload=504459]: Waiting for error group managing segments... Dec 29 14:55:41.267833074 (575h26m41.138916119s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.281060259 (575h26m41.152143305s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.286387874 (575h26m41.157470919s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.287299309 (575h26m41.158382344s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.301675123 (575h26m41.172758169s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.302980108 (575h26m41.174063143s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.315675352 (575h26m41.186758387s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.325229314 (575h26m41.196312349s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.330815401 (575h26m41.201898436s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.341338648 (575h26m41.212421683s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.347929017 (575h26m41.219012062s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.623535403 (575h26m41.494618438s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628522058 (575h26m41.499605093s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628547236 (575h26m41.499630272s) [seg_pos={0 0}, upload=504459]: Segment reached optimal threshold of 13 pieces. Dec 29 14:55:41.628757737 (575h26m41.499840782s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:41.628767966 (575h26m41.499851001s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.628945805 (575h26m41.50002885s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection Dec 29 14:55:41.628954733 (575h26m41.500037778s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629177076 (575h26m41.500260111s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection Dec 29 14:55:41.629199218 (575h26m41.500282273s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629231810 (575h26m41.500314855s) [seg_pos={0 0}, upload=504459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:55:41.629342471 (575h26m41.500425506s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:41.629350496 (575h26m41.500433531s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:41.923681265 (575h26m41.79476431s) [upload=504459]: Done uploading segments... Dec 29 14:55:41.923704099 (575h26m41.794787134s) [upload=504459]: Upload finished. err: <nil> Duration: 6.077815279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.......................X........X.........X......X.......X.] duration: 50.484019418s trace: https://jaeger.dev.storj.io/trace/4d59fb908fcd5e32 FAIL (2025-12-29 14:57:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.201966488 (575h26m33.073049533s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.202316224 (575h26m33.073399259s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203100067 (575h26m33.074183112s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203517372 (575h26m33.074600417s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204138164 (575h26m33.075221209s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204587550 (575h26m33.075670595s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205402070 (575h26m33.076485115s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205695298 (575h26m33.076778333s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.206491314 (575h26m33.077574349s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207058744 (575h26m33.078141789s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207697070 (575h26m33.078780105s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208500180 (575h26m33.079583215s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208704328 (575h26m33.079787364s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.209759517 (575h26m33.080842552s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.210896212 (575h26m33.081979257s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:34.572355890 (575h26m34.443438925s) [upload=504454]: Next returned nil segment Dec 29 14:55:34.572368655 (575h26m34.44345169s) [upload=504454]: Waiting for error group managing segments... Dec 29 14:55:35.103639091 (575h26m34.974722126s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.255917912 (575h26m35.127000967s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.348450017 (575h26m35.219533062s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.519579483 (575h26m35.390662528s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.624938442 (575h26m35.496021487s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.702059762 (575h26m35.573142797s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.875749101 (575h26m35.746832156s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.978308259 (575h26m35.849391294s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.980555457 (575h26m35.851638492s) [noise=true , address=145.79.227.40:4057 , piece_id=YD6LHBJEUJCGRDYDYPSKYMYAXPDI7YXEFEIIX65R4X6I3FIGQBHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.006221009 (575h26m35.877304044s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.015400719 (575h26m35.886483754s) [noise=true , address=145.79.227.30:4781 , piece_id=VGRLYZCMCLDD6YTMENV3FDPUE5T3XZMY37EPZXCI3ZRPRGPESDZA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.034554079 (575h26m35.905637114s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.072374315 (575h26m35.94345735s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127854296 (575h26m35.998937331s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127872321 (575h26m35.998955356s) [seg_pos={0 0}, upload=504454]: Segment reached optimal threshold of 14 pieces. Dec 29 14:55:36.128098612 (575h26m35.999181647s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e7rvSxVxpjoe6zZeXYiVCyDs8sqB4DR1oWN7rGByAdUT9vjp): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.128105936 (575h26m35.999188971s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.128400156 (575h26m35.999483191s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection Dec 29 14:55:36.128406738 (575h26m35.999489773s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129158100 (575h26m36.000241135s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129165283 (575h26m36.000248318s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129232621 (575h26m36.000315656s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129239735 (575h26m36.00032277s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129261206 (575h26m36.000344241s) [seg_pos={0 0}, upload=504454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:55:36.129337591 (575h26m36.000420626s) [upload=504454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:36.129342060 (575h26m36.000425095s) [upload=504454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:36.429197408 (575h26m36.300280453s) [upload=504454]: Done uploading segments... Dec 29 14:55:36.429249858 (575h26m36.300332904s) [upload=504454]: Upload finished. err: <nil> Duration: 50.48403583s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 15.305269207s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:57:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 15.305292311s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.X..............................XXX...........XX.X..........] duration: 5.80351074s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:57:09 UTC) orbiter warning: context deadline exceeded Duration: 5.803537712s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXX] duration: 5.001812162s trace: https://jaeger.dev.storj.io/trace/7089f074b8ef247e FAIL (2025-12-29 14:57:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00184226s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.XX.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX....] duration: 20.625218087s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:56:58 UTC) orbiter warning: context deadline exceeded Duration: 20.625242825s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX.......................X........X.........X......X.......X] duration: 1m0.002366233s trace: https://jaeger.dev.storj.io/trace/4d59fb908fcd5e32 FAIL (2025-12-29 14:56:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.201966488 (575h26m33.073049533s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.202316224 (575h26m33.073399259s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203100067 (575h26m33.074183112s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.203517372 (575h26m33.074600417s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204138164 (575h26m33.075221209s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.204587550 (575h26m33.075670595s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205402070 (575h26m33.076485115s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.205695298 (575h26m33.076778333s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.206491314 (575h26m33.077574349s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207058744 (575h26m33.078141789s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.207697070 (575h26m33.078780105s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208500180 (575h26m33.079583215s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.208704328 (575h26m33.079787364s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.209759517 (575h26m33.080842552s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:33.210896212 (575h26m33.081979257s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Uploading piece... Dec 29 14:55:34.572355890 (575h26m34.443438925s) [upload=504454]: Next returned nil segment Dec 29 14:55:34.572368655 (575h26m34.44345169s) [upload=504454]: Waiting for error group managing segments... Dec 29 14:55:35.103639091 (575h26m34.974722126s) [noise=true , address=145.79.227.16:4928 , piece_id=MLFMJHKY47M5GS4LRAKW5EHXNX5THXFYVXTDCAT465I4XD6YP3UQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.255917912 (575h26m35.127000967s) [noise=true , address=77.90.31.17:4782 , piece_id=WC3J33AROYRB3UEDOAYU4D5VI2CMVLDDJZBG4QPKYZJ4BZBCJDWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.348450017 (575h26m35.219533062s) [noise=true , address=45.91.116.10:4115 , piece_id=EBCTAULLOMWW4U3RV2NBYEHSTJX2XMSROJJL74HBPPBTWUQHQYAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.519579483 (575h26m35.390662528s) [noise=true , address=77.90.31.12:4623 , piece_id=J3JDRSH4ZLNYXINDPCAHZJG4RHSFWT7UMHFC2GUYZOXQFEQUNKAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.624938442 (575h26m35.496021487s) [noise=true , address=77.90.31.21:4884 , piece_id=U4IJVM2AY7LCCKO7Z3STHUFC2A3VCAYW6JOTZ3S5ECHGVHQKNAFA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.702059762 (575h26m35.573142797s) [noise=true , address=45.91.116.11:4137 , piece_id=JVBBB45PWQ7C7X7NM3KUH4YTPJOTTRB5TA2IE433JK5TR3GGUOHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.875749101 (575h26m35.746832156s) [noise=true , address=193.149.164.143:4694 , piece_id=WUXQGOYM7RZPUGLMY2G4ZPENSYF2SYOQZ7WD5YBVRVTE6GPQHWAA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.978308259 (575h26m35.849391294s) [noise=true , address=77.90.31.11:4575 , piece_id=N7I4UAJK7CAAMWB52CSICX737BX4VBYXKWEGR43UZWKIGFYKFD5A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:35.980555457 (575h26m35.851638492s) [noise=true , address=145.79.227.40:4057 , piece_id=YD6LHBJEUJCGRDYDYPSKYMYAXPDI7YXEFEIIX65R4X6I3FIGQBHA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.006221009 (575h26m35.877304044s) [noise=true , address=77.90.31.16:4742 , piece_id=O6LCOGLTDIHQNBMZOBD3IMAYQW42EDPBOJBDH7UK7ZCFFG7BXMOA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.015400719 (575h26m35.886483754s) [noise=true , address=145.79.227.30:4781 , piece_id=VGRLYZCMCLDD6YTMENV3FDPUE5T3XZMY37EPZXCI3ZRPRGPESDZA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.034554079 (575h26m35.905637114s) [noise=true , address=145.79.227.44:4212 , piece_id=2F2SP3ES2MMOC62YNJD35DHB5SSM2JV6IL3EUHKMGU2OEIULCWWQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.072374315 (575h26m35.94345735s) [noise=true , address=145.79.227.29:4970 , piece_id=LR2HAMYYMM2NG4LDDQTYJRW66OHQ463V5WQJHA2CXZQEYA3LIFDQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127854296 (575h26m35.998937331s) [noise=true , address=145.79.227.34:4928 , piece_id=M2LY2OTLBSXOISYXVUJPXIXDQNOQBVRB2BAGPPH6NFOCDNSEFCEA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: <nil> Dec 29 14:55:36.127872321 (575h26m35.998955356s) [seg_pos={0 0}, upload=504454]: Segment reached optimal threshold of 14 pieces. Dec 29 14:55:36.128098612 (575h26m35.999181647s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e7rvSxVxpjoe6zZeXYiVCyDs8sqB4DR1oWN7rGByAdUT9vjp): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.128105936 (575h26m35.999188971s) [noise=true , address=45.91.116.11:4125 , piece_id=A2I2XXTA3374TDVYE3NCUEFHKLFMIP6PWKIOAFR5WD7IBRJW4SUQ, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.128400156 (575h26m35.999483191s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46736->64.49.13.169:4713: use of closed network connection Dec 29 14:55:36.128406738 (575h26m35.999489773s) [noise=true , address=64.49.13.169:4713 , piece_id=2HKRYH2RNG4MDMJ6Q2NNFHXMAKO2LIUXH3DYZNXHMHM76IGUC55Q, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129158100 (575h26m36.000241135s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129165283 (575h26m36.000248318s) [noise=true , address=45.91.116.9:4279 , piece_id=7P2GEBJ2CVX75NXXLZ3JVC63CB27OGRK3ME3ITZNRS5HHLRMSS3A, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129232621 (575h26m36.000315656s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:36.129239735 (575h26m36.00032277s) [noise=true , address=145.79.227.33:4899 , piece_id=5LABAR4DR7GPVQJ74XB4LR5HTVFWHFMLNQHMO4CYKN6R2R4SSTMA, seg_pos={0 0}, upload=504454]: Long tail cancellation due to optimal threshold Dec 29 14:55:36.129261206 (575h26m36.000344241s) [seg_pos={0 0}, upload=504454]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:55:36.129337591 (575h26m36.000420626s) [upload=504454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:36.129342060 (575h26m36.000425095s) [upload=504454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:36.429197408 (575h26m36.300280453s) [upload=504454]: Done uploading segments... Dec 29 14:55:36.429249858 (575h26m36.300332904s) [upload=504454]: Upload finished. err: <nil> Duration: 1m0.002376602s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 4.563085176s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:56:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 4.563115223s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration: 5.001274155s trace: https://jaeger.dev.storj.io/trace/7f9cb20d5b6983ac FAIL (2025-12-29 14:56:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00129805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..X..............................XXX...........XX.X.........] duration: 19.488147296s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:56:23 UTC) orbiter warning: context deadline exceeded Duration: 19.488177372s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.XXXXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX.] duration: 5.983959156s trace: https://jaeger.dev.storj.io/trace/6eb6a4a5ee96c354 FAIL (2025-12-29 14:56:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 29 14:55:39.029548108 (575h26m38.900631153s) [upload=504459]: Got next segment. Inline: false Dec 29 14:55:39.029620336 (575h26m38.900703381s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:55:39.029628863 (575h26m38.900711898s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:55:39.274058943 (575h26m39.145141978s) [seg_pos={0 0}, upload=504459]: Begin upload segment... Dec 29 14:55:39.274103809 (575h26m39.145186844s) [seg_pos={0 0}, upload=504459]: Done begin upload segment. Dec 29 14:55:39.274118707 (575h26m39.145201752s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274171288 (575h26m39.145254323s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274211514 (575h26m39.145294549s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275556354 (575h26m39.146639389s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275649141 (575h26m39.146732176s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276045065 (575h26m39.14712811s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276399980 (575h26m39.147483016s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276513698 (575h26m39.147596733s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276936142 (575h26m39.148019187s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.277332496 (575h26m39.148415531s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.286003968 (575h26m39.157087003s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287295517 (575h26m39.158378553s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287744863 (575h26m39.158827898s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.288054283 (575h26m39.159137328s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.289285316 (575h26m39.160368361s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.290576334 (575h26m39.16165938s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:40.994006767 (575h26m40.865089802s) [upload=504459]: Next returned nil segment Dec 29 14:55:40.994030793 (575h26m40.865113828s) [upload=504459]: Waiting for error group managing segments... Dec 29 14:55:41.267833074 (575h26m41.138916119s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.281060259 (575h26m41.152143305s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.286387874 (575h26m41.157470919s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.287299309 (575h26m41.158382344s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.301675123 (575h26m41.172758169s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.302980108 (575h26m41.174063143s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.315675352 (575h26m41.186758387s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.325229314 (575h26m41.196312349s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.330815401 (575h26m41.201898436s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.341338648 (575h26m41.212421683s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.347929017 (575h26m41.219012062s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.623535403 (575h26m41.494618438s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628522058 (575h26m41.499605093s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628547236 (575h26m41.499630272s) [seg_pos={0 0}, upload=504459]: Segment reached optimal threshold of 13 pieces. Dec 29 14:55:41.628757737 (575h26m41.499840782s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:41.628767966 (575h26m41.499851001s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.628945805 (575h26m41.50002885s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection Dec 29 14:55:41.628954733 (575h26m41.500037778s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629177076 (575h26m41.500260111s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection Dec 29 14:55:41.629199218 (575h26m41.500282273s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629231810 (575h26m41.500314855s) [seg_pos={0 0}, upload=504459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:55:41.629342471 (575h26m41.500425506s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:41.629350496 (575h26m41.500433531s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:41.923681265 (575h26m41.79476431s) [upload=504459]: Done uploading segments... Dec 29 14:55:41.923704099 (575h26m41.794787134s) [upload=504459]: Upload finished. err: <nil> Duration: 5.983973063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX.XXXXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXXX] duration: 30.002291233s trace: https://jaeger.dev.storj.io/trace/6eb6a4a5ee96c354 FAIL (2025-12-29 14:56:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 29 14:55:39.029548108 (575h26m38.900631153s) [upload=504459]: Got next segment. Inline: false Dec 29 14:55:39.029620336 (575h26m38.900703381s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:55:39.029628863 (575h26m38.900711898s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:55:39.274058943 (575h26m39.145141978s) [seg_pos={0 0}, upload=504459]: Begin upload segment... Dec 29 14:55:39.274103809 (575h26m39.145186844s) [seg_pos={0 0}, upload=504459]: Done begin upload segment. Dec 29 14:55:39.274118707 (575h26m39.145201752s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274171288 (575h26m39.145254323s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.274211514 (575h26m39.145294549s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275556354 (575h26m39.146639389s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.275649141 (575h26m39.146732176s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276045065 (575h26m39.14712811s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276399980 (575h26m39.147483016s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276513698 (575h26m39.147596733s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.276936142 (575h26m39.148019187s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.277332496 (575h26m39.148415531s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.286003968 (575h26m39.157087003s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287295517 (575h26m39.158378553s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.287744863 (575h26m39.158827898s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.288054283 (575h26m39.159137328s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.289285316 (575h26m39.160368361s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:39.290576334 (575h26m39.16165938s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Uploading piece... Dec 29 14:55:40.994006767 (575h26m40.865089802s) [upload=504459]: Next returned nil segment Dec 29 14:55:40.994030793 (575h26m40.865113828s) [upload=504459]: Waiting for error group managing segments... Dec 29 14:55:41.267833074 (575h26m41.138916119s) [noise=true , address=145.79.227.11:4730 , piece_id=NKUGTTRCYOCCMGZJFNTLTUOK4UYIPGNMY7EGNAJW4HM2PIAWSFNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.281060259 (575h26m41.152143305s) [noise=true , address=145.79.227.8:4331 , piece_id=HAXE6HPCGPQSLBW5MJQYHIU7YHWREKURBXMY2DD64MNOIEH2VZIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.286387874 (575h26m41.157470919s) [noise=true , address=145.79.227.39:4014 , piece_id=URWRCGMAMRRK4AEL3UPKMWUZQ7J4VMBYNASYUBLA2UPBSXIBI5SA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.287299309 (575h26m41.158382344s) [noise=true , address=145.79.227.2:4113 , piece_id=IBT7EQKQ3FDCUYOYZHMTYLH6FGIRZB6X7BBNK3RDERGE7RBSZARQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.301675123 (575h26m41.172758169s) [noise=true , address=145.79.227.4:4186 , piece_id=J75DD36TUUXYCBFSU2MUXX52RGNROWEGLQ56QZ6TA3IDUMBRJMMQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.302980108 (575h26m41.174063143s) [noise=true , address=145.79.227.42:4121 , piece_id=A526LYOMKKNCXMUUCVS74T2SAEYJQJRTURFXO4V672DQKGA7VNLQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.315675352 (575h26m41.186758387s) [noise=true , address=145.79.227.15:4875 , piece_id=R4O2S7Y7T5BC2XM5HRU52TKZKD2C5P4RZE3IAEGPOGFWY5BLQXNA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.325229314 (575h26m41.196312349s) [noise=true , address=145.79.227.14:4838 , piece_id=CK6U6DYSR52J4CKJZKX2XSSQTICIUKPAZ7IXWBMJOSQSD6E7AMUA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.330815401 (575h26m41.201898436s) [noise=true , address=145.79.227.37:4567 , piece_id=4XRY6IAT3S626RT4MYHIJD4FDOIC5UBGHHOXELDXJVW42OQEC4PA, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.341338648 (575h26m41.212421683s) [noise=true , address=145.79.227.50:4373 , piece_id=KPBI2X3XONHIDT2YWKGPFCJ4VKPVXWNA5EXS54MHJX2KXY54BIYQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.347929017 (575h26m41.219012062s) [noise=true , address=145.79.227.48:4301 , piece_id=A53FKOOENFES3AJP3EG2YI3DVT5NI2CR3JX5XGBZFXOZHP5SRCVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.623535403 (575h26m41.494618438s) [noise=true , address=45.91.116.8:4047 , piece_id=BWBTFL3CZE4W5V5SSFIW5IH4SZZOF4KF2J5QW4B3L2OPJ3PS47RQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628522058 (575h26m41.499605093s) [noise=true , address=45.91.116.12:4225 , piece_id=IG43NMNKNP6HOWRY5J2MXE7TGS2UHZWRIXIBQKMKARFIN2BC4MIQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: <nil> Dec 29 14:55:41.628547236 (575h26m41.499630272s) [seg_pos={0 0}, upload=504459]: Segment reached optimal threshold of 13 pieces. Dec 29 14:55:41.628757737 (575h26m41.499840782s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:41.628767966 (575h26m41.499851001s) [noise=true , address=77.90.31.11:4554 , piece_id=N72Z3JBYAH6YCX2TYRZGVJ6ONKU4UM6UCPWJY6YZO5F2WQZQUH5A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.628945805 (575h26m41.50002885s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51478->45.91.116.6:4974: use of closed network connection Dec 29 14:55:41.628954733 (575h26m41.500037778s) [noise=true , address=45.91.116.6:4974 , piece_id=OHQTJHM4VK5WBSGQIO3HH45SVXW47QKWOWXETVYH5MBTRV6YVOVQ, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629177076 (575h26m41.500260111s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39022->45.91.116.10:4082: use of closed network connection Dec 29 14:55:41.629199218 (575h26m41.500282273s) [noise=true , address=45.91.116.10:4082 , piece_id=6VRJ2RYGQ2MCHEWAONGUFJ3CY6QTFTAVMDO52CN6REKNGLY6GE7A, seg_pos={0 0}, upload=504459]: Long tail cancellation due to optimal threshold Dec 29 14:55:41.629231810 (575h26m41.500314855s) [seg_pos={0 0}, upload=504459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:55:41.629342471 (575h26m41.500425506s) [upload=504459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:41.629350496 (575h26m41.500433531s) [upload=504459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:41.923681265 (575h26m41.79476431s) [upload=504459]: Done uploading segments... Dec 29 14:55:41.923704099 (575h26m41.794787134s) [upload=504459]: Upload finished. err: <nil> Duration: 30.002311191s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X.XX.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX...] duration: 24.046122938s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:56:02 UTC) orbiter warning: context deadline exceeded Duration: 24.04615444s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 12.660765142s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:55:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.660805951s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...X..............................XXX...........XX.X........] duration: 14.71583218s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:55:49 UTC) orbiter warning: context deadline exceeded Duration: 14.715849082s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....X..............................XXX...........XX.X.......] duration: 24.374200493s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:55:34 UTC) orbiter warning: context deadline exceeded Duration: 24.37421916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXX.XXXXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XXX] duration: 30.003758646s trace: https://jaeger.dev.storj.io/trace/4c73a336acae94fa FAIL (2025-12-29 14:55:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 4HPV2JOWHKVFXJ3EOHRZIPJ4O62VLLRMJ5ORLOXZCJIF3C7ZAUSA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: N3S7EKTQRSV5SHP5Q3ENWPQZFSS5DSCSFVZTP6SSQOYNRF5YC5DQ) context deadline exceeded ======LOGS====== Dec 29 14:55:03.959698421 (575h26m3.830781466s) [upload=504448]: Starting upload Dec 29 14:55:03.959802840 (575h26m3.830885905s) [upload=504448]: Done starting upload Dec 29 14:55:03.959863465 (575h26m3.83094652s) [upload=504448]: Uploading segments... Dec 29 14:55:03.961631893 (575h26m3.832714938s) [upload=504448]: Got next segment. Inline: false Dec 29 14:55:03.961702327 (575h26m3.832785372s) [upload=504448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:55:03.961709731 (575h26m3.832792776s) [upload=504448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:55:04.208532568 (575h26m4.079615613s) [seg_pos={0 0}, upload=504448]: Begin upload segment... Dec 29 14:55:04.208595909 (575h26m4.079678954s) [seg_pos={0 0}, upload=504448]: Done begin upload segment. Dec 29 14:55:04.208647196 (575h26m4.079730241s) [noise=true , address=145.79.227.21:4475 , piece_id=R7FLUM3VLMPXTTZQNPLOIDIY3YBMDA2UPEPFYQ2TP6NCISYJFU2Q, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.208682764 (575h26m4.079765809s) [noise=true , address=145.79.227.12:4767 , piece_id=6BXFFR6TJ4N56FKTHH2C7JOLDHLGIUDVQVP7SLKLPNJMPE3BFYZQ, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.209874353 (575h26m4.080957398s) [noise=true , address=145.79.227.15:4874 , piece_id=Z5L5ZA5KCGSC4HUEAR5HEHJ6IMNW2KVN4RH6NGZJQHG7C2Y56JDA, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.210535933 (575h26m4.081618978s) [noise=true , address=145.79.227.2:4113 , piece_id=4TI2LS5QQ3C3ZMO3HAC3LTS4D2ZF5D3TQYH37ZNNQNXZZDLBLJRQ, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.211133350 (575h26m4.082216396s) [noise=true , address=145.79.227.38:4977 , piece_id=WI2GTXS4DZFT5BJZAXKYBIN24UZ23INNLJHDKNLMUZJ2KVYNZ6LQ, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.211744423 (575h26m4.082827479s) [noise=true , address=145.79.227.41:4085 , piece_id=MELHI4PC7B4UCEAPRQFW75DWFPMOSSFC5B4RBTAAXIFXPTJ2GLLQ, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.212286987 (575h26m4.083370032s) [noise=true , address=145.79.227.5:4222 , piece_id=KOZEL54OJADYFAYOLD4R7DNVPWBNUGVUNYVKCEB7VEU34SLC3OJA, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.212895255 (575h26m4.0839783s) [noise=true , address=45.91.116.13:4153 , piece_id=VVBB46GNOETMMCP6IKVKPLOUIAP4FLGB7FVFT2FR4YHJ2CLJPPOQ, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.212906537 (575h26m4.083989582s) [noise=true , address=45.91.116.11:4118 , piece_id=63QU6DC7XLZVA6XU5YJG36HDIVKWUTXK5VMLLVUZP5GGZRUAH7FQ, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.213745414 (575h26m4.084828459s) [noise=true , address=145.79.227.48:4301 , piece_id=QX7SMZ7VZMNPABMQK3WXLP2FIMFMRLQU6QIYZHTTXPIRNIEGUHNA, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.214047710 (575h26m4.085130755s) [noise=true , address=145.79.227.51:4410 , piece_id=3UJH7LGF5CI4TIKPPIE5V2Q634GGX2UAPW2BU3UNIJYLH5ME4PUQ, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.214430639 (575h26m4.085513724s) [noise=true , address=77.90.31.10:4519 , piece_id=4HPV2JOWHKVFXJ3EOHRZIPJ4O62VLLRMJ5ORLOXZCJIF3C7ZAUSA, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.214891927 (575h26m4.085974973s) [noise=true , address=45.91.116.7:4009 , piece_id=6GWMOSCI53L2F46VSEUOYWUMTGXMQ2BPXVYBB5WZU4JF4BWQ662A, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.223215096 (575h26m4.094298141s) [noise=true , address=145.79.227.39:4014 , piece_id=U72AN2VLCYJ363LLN43KHQ7J5Q5ALJXLN566BGJNALSLXGPNCTAQ, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.233788249 (575h26m4.104871305s) [noise=true , address=145.79.227.42:4123 , piece_id=TVA5YUGIFL2XIJC54V6W4D6NRLGSDYKHYKZIWO6ZMXHJSBGIAQUA, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:04.235772328 (575h26m4.106855393s) [noise=true , address=45.91.116.12:4227 , piece_id=N3S7EKTQRSV5SHP5Q3ENWPQZFSS5DSCSFVZTP6SSQOYNRF5YC5DQ, seg_pos={0 0}, upload=504448]: Uploading piece... Dec 29 14:55:05.883591109 (575h26m5.754674144s) [upload=504448]: Next returned nil segment Dec 29 14:55:05.883610706 (575h26m5.754693741s) [upload=504448]: Waiting for error group managing segments... Dec 29 14:55:06.210727585 (575h26m6.08181062s) [noise=true , address=145.79.227.38:4977 , piece_id=WI2GTXS4DZFT5BJZAXKYBIN24UZ23INNLJHDKNLMUZJ2KVYNZ6LQ, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.211736065 (575h26m6.0828191s) [noise=true , address=145.79.227.5:4222 , piece_id=KOZEL54OJADYFAYOLD4R7DNVPWBNUGVUNYVKCEB7VEU34SLC3OJA, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.215241430 (575h26m6.086324465s) [noise=true , address=145.79.227.12:4767 , piece_id=6BXFFR6TJ4N56FKTHH2C7JOLDHLGIUDVQVP7SLKLPNJMPE3BFYZQ, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.215520892 (575h26m6.086603937s) [noise=true , address=145.79.227.15:4874 , piece_id=Z5L5ZA5KCGSC4HUEAR5HEHJ6IMNW2KVN4RH6NGZJQHG7C2Y56JDA, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.215862412 (575h26m6.086945447s) [noise=true , address=145.79.227.2:4113 , piece_id=4TI2LS5QQ3C3ZMO3HAC3LTS4D2ZF5D3TQYH37ZNNQNXZZDLBLJRQ, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.229497064 (575h26m6.10058011s) [noise=true , address=145.79.227.41:4085 , piece_id=MELHI4PC7B4UCEAPRQFW75DWFPMOSSFC5B4RBTAAXIFXPTJ2GLLQ, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.230292259 (575h26m6.101375304s) [noise=true , address=145.79.227.48:4301 , piece_id=QX7SMZ7VZMNPABMQK3WXLP2FIMFMRLQU6QIYZHTTXPIRNIEGUHNA, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.232565467 (575h26m6.103648502s) [noise=true , address=145.79.227.21:4475 , piece_id=R7FLUM3VLMPXTTZQNPLOIDIY3YBMDA2UPEPFYQ2TP6NCISYJFU2Q, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.233253187 (575h26m6.104336222s) [noise=true , address=145.79.227.39:4014 , piece_id=U72AN2VLCYJ363LLN43KHQ7J5Q5ALJXLN566BGJNALSLXGPNCTAQ, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.238537549 (575h26m6.109620584s) [noise=true , address=145.79.227.51:4410 , piece_id=3UJH7LGF5CI4TIKPPIE5V2Q634GGX2UAPW2BU3UNIJYLH5ME4PUQ, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.254435812 (575h26m6.125518847s) [noise=true , address=145.79.227.42:4123 , piece_id=TVA5YUGIFL2XIJC54V6W4D6NRLGSDYKHYKZIWO6ZMXHJSBGIAQUA, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.466460639 (575h26m6.337543684s) [noise=true , address=45.91.116.12:4227 , piece_id=N3S7EKTQRSV5SHP5Q3ENWPQZFSS5DSCSFVZTP6SSQOYNRF5YC5DQ, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.672225184 (575h26m6.543308219s) [noise=true , address=77.90.31.10:4519 , piece_id=4HPV2JOWHKVFXJ3EOHRZIPJ4O62VLLRMJ5ORLOXZCJIF3C7ZAUSA, seg_pos={0 0}, upload=504448]: Done uploading piece. err: <nil> Dec 29 14:55:06.672249982 (575h26m6.543333027s) [seg_pos={0 0}, upload=504448]: Segment reached optimal threshold of 13 pieces. Dec 29 14:55:06.672534102 (575h26m6.543617148s) [noise=true , address=45.91.116.11:4118 , piece_id=63QU6DC7XLZVA6XU5YJG36HDIVKWUTXK5VMLLVUZP5GGZRUAH7FQ, seg_pos={0 0}, upload=504448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:06.672542779 (575h26m6.543625814s) [noise=true , address=45.91.116.11:4118 , piece_id=63QU6DC7XLZVA6XU5YJG36HDIVKWUTXK5VMLLVUZP5GGZRUAH7FQ, seg_pos={0 0}, upload=504448]: Long tail cancellation due to optimal threshold Dec 29 14:55:06.672657548 (575h26m6.543740593s) [noise=true , address=45.91.116.13:4153 , piece_id=VVBB46GNOETMMCP6IKVKPLOUIAP4FLGB7FVFT2FR4YHJ2CLJPPOQ, seg_pos={0 0}, upload=504448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 29 14:55:06.672675281 (575h26m6.543758326s) [noise=true , address=45.91.116.13:4153 , piece_id=VVBB46GNOETMMCP6IKVKPLOUIAP4FLGB7FVFT2FR4YHJ2CLJPPOQ, seg_pos={0 0}, upload=504448]: Long tail cancellation due to optimal threshold Dec 29 14:55:06.672816911 (575h26m6.543899956s) [noise=true , address=45.91.116.7:4009 , piece_id=6GWMOSCI53L2F46VSEUOYWUMTGXMQ2BPXVYBB5WZU4JF4BWQ662A, seg_pos={0 0}, upload=504448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:41286->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:41286->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41286->45.91.116.7:4009: use of closed network connection Dec 29 14:55:06.672856847 (575h26m6.543939902s) [noise=true , address=45.91.116.7:4009 , piece_id=6GWMOSCI53L2F46VSEUOYWUMTGXMQ2BPXVYBB5WZU4JF4BWQ662A, seg_pos={0 0}, upload=504448]: Long tail cancellation due to optimal threshold Dec 29 14:55:06.672877046 (575h26m6.543960081s) [seg_pos={0 0}, upload=504448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:55:06.672951948 (575h26m6.544034993s) [upload=504448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:55:06.672958500 (575h26m6.544041535s) [upload=504448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:55:06.976443511 (575h26m6.847526556s) [upload=504448]: Done uploading segments... Dec 29 14:55:06.976467496 (575h26m6.847550541s) [upload=504448]: Upload finished. err: <nil> Duration: 30.00377102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX.......................X........X.........X......X.......] duration: 54.251314684s trace: https://jaeger.dev.storj.io/trace/6478a0d64487577a FAIL (2025-12-29 14:55:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4771 , piece_id=Q6LDYGDGYLATAH7JOZB2OUO2IYB552S4KQXLAEBQBP2IQJB3DEHA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.251070129 (575h19m11.122153164s) [noise=true , address=77.90.31.19:4863 , piece_id=PEP4JSYV5D7FMKQ7C3YKEDQDZRSEPWG5JDSNHI2ZLRADV2UCL57A, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.251437850 (575h19m11.122520885s) [noise=true , address=145.79.227.33:4886 , piece_id=DUAYROBYHSTD7JYTY4NG4ICWZL4BZWYXBJ3REKXE266LEMJSNNMA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.251817152 (575h19m11.122900197s) [noise=true , address=77.90.31.14:4682 , piece_id=CDHWFERWSBUZDKR4CIWQXU542WKXDWXHUSLRJAHQSBUSDOXOVUXA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.252173631 (575h19m11.123256666s) [noise=true , address=145.79.227.29:4965 , piece_id=DDQ5OI3ZTGJW72S7BG4JBNZTFHCEBDUOXWVH4ZXDB6YLXQXTOGRQ, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.252530560 (575h19m11.123613595s) [noise=true , address=64.49.13.170:4723 , piece_id=LFRH7RBK3C5WZSY5BBRXQJBWEMR7E5U4CNN6VH4Q5DLNMXHOZ6OA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.252547924 (575h19m11.123630969s) [noise=true , address=145.79.227.40:4072 , piece_id=DEQR47CTRJ3FGAJZI5WNQOSYZWYYYQ7A76MWD3GKWYH2D62A67WQ, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.252922045 (575h19m11.12400509s) [noise=true , address=77.90.31.18:4838 , piece_id=2EZKOWKFXCIP5XNLQ34ZH4U2TZVNVON3HXMT6LBWS2BOVAZIUM6Q, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.253001848 (575h19m11.124084893s) [noise=true , address=145.79.227.13:4806 , piece_id=NMBGJGVJRCBDXBUQRGY3HIWN2GLR5G442WVVAOMCPVZTNCAHL5XA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.253314153 (575h19m11.124397188s) [noise=true , address=77.90.31.13:4643 , piece_id=XCU3O7UI2ZLWTFUUFL2VYIFNTZYAVXQQZMEEQ27UIRQQUOPDVQSA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.253496950 (575h19m11.124579995s) [noise=true , address=145.79.227.41:4091 , piece_id=NVY2H5JKILJPY7JI35DNVPMECWZARQ4RFYZFONDWGCVM4MVEJAZQ, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.253995289 (575h19m11.125078334s) [noise=true , address=145.79.227.3:4156 , piece_id=XZODCLD2FVVPCGS7NXAJWUADGCRFZHRH4SJLQWIYZTX52R5ENJ7A, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.254466236 (575h19m11.125549281s) [noise=true , address=64.49.13.169:4707 , piece_id=V5M4DJFXHHFJOUYPUEGLZ4SF7GKNU7W3DTSLXZXLVO2WPCXDII3A, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.254911644 (575h19m11.125994689s) [noise=true , address=193.149.164.143:4695 , piece_id=LRP4QCSTS4LM5GR2XCCSDF5FF4JIOZSUVFWMWBQDSUNJLWN4F2KQ, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.255366950 (575h19m11.126449996s) [noise=true , address=45.91.116.13:4173 , piece_id=OSVSBH5XSCDPJRPHRMZBO33FKG67ABSM3Z5TUTV2FPFXK5A7IPSA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.255817408 (575h19m11.126900453s) [noise=true , address=145.79.227.43:4177 , piece_id=AVQC3VFEY63SR5BV4OZI4GHHZXWLNELTOUWVLZNL5V5OFAF7FCSQ, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:12.551291333 (575h19m12.422374368s) [upload=504353]: Next returned nil segment Dec 29 14:48:12.551313084 (575h19m12.422396129s) [upload=504353]: Waiting for error group managing segments... Dec 29 14:48:12.879067587 (575h19m12.750150632s) [noise=true , address=64.49.13.170:4723 , piece_id=LFRH7RBK3C5WZSY5BBRXQJBWEMR7E5U4CNN6VH4Q5DLNMXHOZ6OA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.112148518 (575h19m12.983231563s) [noise=true , address=145.79.227.41:4091 , piece_id=NVY2H5JKILJPY7JI35DNVPMECWZARQ4RFYZFONDWGCVM4MVEJAZQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.112578336 (575h19m12.983661401s) [noise=true , address=145.79.227.13:4806 , piece_id=NMBGJGVJRCBDXBUQRGY3HIWN2GLR5G442WVVAOMCPVZTNCAHL5XA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.122458630 (575h19m12.993541675s) [noise=true , address=145.79.227.29:4965 , piece_id=DDQ5OI3ZTGJW72S7BG4JBNZTFHCEBDUOXWVH4ZXDB6YLXQXTOGRQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.329124643 (575h19m13.200207678s) [noise=true , address=77.90.31.14:4682 , piece_id=CDHWFERWSBUZDKR4CIWQXU542WKXDWXHUSLRJAHQSBUSDOXOVUXA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.336139901 (575h19m13.207222936s) [noise=true , address=77.90.31.13:4643 , piece_id=XCU3O7UI2ZLWTFUUFL2VYIFNTZYAVXQQZMEEQ27UIRQQUOPDVQSA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.427538136 (575h19m13.298621181s) [noise=true , address=77.90.31.18:4838 , piece_id=2EZKOWKFXCIP5XNLQ34ZH4U2TZVNVON3HXMT6LBWS2BOVAZIUM6Q, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.628870834 (575h19m13.499953869s) [noise=true , address=45.91.116.13:4173 , piece_id=OSVSBH5XSCDPJRPHRMZBO33FKG67ABSM3Z5TUTV2FPFXK5A7IPSA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.956600930 (575h19m13.827683965s) [noise=true , address=145.79.227.3:4156 , piece_id=XZODCLD2FVVPCGS7NXAJWUADGCRFZHRH4SJLQWIYZTX52R5ENJ7A, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.982179565 (575h19m13.8532626s) [noise=true , address=193.149.164.143:4695 , piece_id=LRP4QCSTS4LM5GR2XCCSDF5FF4JIOZSUVFWMWBQDSUNJLWN4F2KQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:14.019680373 (575h19m13.890763408s) [noise=true , address=145.79.227.34:4936 , piece_id=R5UZN7BR2YFU7JMGIGZUEXSMS6ECMGS66ZQZKY3INVLDSNYHQ4OQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:14.034961690 (575h19m13.906044735s) [noise=true , address=145.79.227.43:4177 , piece_id=AVQC3VFEY63SR5BV4OZI4GHHZXWLNELTOUWVLZNL5V5OFAF7FCSQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:14.090555199 (575h19m13.961638234s) [noise=true , address=145.79.227.44:4210 , piece_id=MCA3TWGDIW76ZSTBFZRCI7D7NSFC6IQ3E7NEG4I3ZR7OY3M7VCIQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:14.130988049 (575h19m14.002071094s) [noise=true , address=145.79.227.30:4771 , piece_id=Q6LDYGDGYLATAH7JOZB2OUO2IYB552S4KQXLAEBQBP2IQJB3DEHA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:14.131012024 (575h19m14.002095089s) [seg_pos={0 0}, upload=504353]: Segment reached optimal threshold of 14 pieces. Dec 29 14:48:14.131247373 (575h19m14.002330408s) [noise=true , address=145.79.227.33:4886 , piece_id=DUAYROBYHSTD7JYTY4NG4ICWZL4BZWYXBJ3REKXE266LEMJSNNMA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 29 14:48:14.131256830 (575h19m14.002339895s) [noise=true , address=145.79.227.33:4886 , piece_id=DUAYROBYHSTD7JYTY4NG4ICWZL4BZWYXBJ3REKXE266LEMJSNNMA, seg_pos={0 0}, upload=504353]: Long tail cancellation due to optimal threshold Dec 29 14:48:14.131360277 (575h19m14.002443312s) [noise=true , address=64.49.13.169:4707 , piece_id=V5M4DJFXHHFJOUYPUEGLZ4SF7GKNU7W3DTSLXZXLVO2WPCXDII3A, seg_pos={0 0}, upload=504353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:40890->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:40890->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40890->64.49.13.169:4707: use of closed network connection Dec 29 14:48:14.131381588 (575h19m14.002464653s) [noise=true , address=64.49.13.169:4707 , piece_id=V5M4DJFXHHFJOUYPUEGLZ4SF7GKNU7W3DTSLXZXLVO2WPCXDII3A, seg_pos={0 0}, upload=504353]: Long tail cancellation due to optimal threshold Dec 29 14:48:14.131587190 (575h19m14.002670275s) [noise=true , address=77.90.31.19:4863 , piece_id=PEP4JSYV5D7FMKQ7C3YKEDQDZRSEPWG5JDSNHI2ZLRADV2UCL57A, seg_pos={0 0}, upload=504353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF Dec 29 14:48:14.131596728 (575h19m14.002679773s) [noise=true , address=77.90.31.19:4863 , piece_id=PEP4JSYV5D7FMKQ7C3YKEDQDZRSEPWG5JDSNHI2ZLRADV2UCL57A, seg_pos={0 0}, upload=504353]: Long tail cancellation due to optimal threshold Dec 29 14:48:14.131674396 (575h19m14.002757441s) [noise=true , address=145.79.227.40:4072 , piece_id=DEQR47CTRJ3FGAJZI5WNQOSYZWYYYQ7A76MWD3GKWYH2D62A67WQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 29 14:48:14.131704453 (575h19m14.002787498s) [noise=true , address=145.79.227.40:4072 , piece_id=DEQR47CTRJ3FGAJZI5WNQOSYZWYYYQ7A76MWD3GKWYH2D62A67WQ, seg_pos={0 0}, upload=504353]: Long tail cancellation due to optimal threshold Dec 29 14:48:14.131739099 (575h19m14.002822134s) [seg_pos={0 0}, upload=504353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:48:14.131846874 (575h19m14.002929929s) [upload=504353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:48:14.131873054 (575h19m14.002956089s) [upload=504353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:48:14.446199770 (575h19m14.317282815s) [upload=504353]: Done uploading segments... Dec 29 14:48:14.446218866 (575h19m14.317301901s) [upload=504353]: Upload finished. err: <nil> Duration: 54.25134429s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X.XX.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX..] duration: 21.742307579s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:55:27 UTC) orbiter warning: context deadline exceeded Duration: 21.742335181s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration: 5.000874734s trace: https://jaeger.dev.storj.io/trace/30f6ee6c9b099c10 FAIL (2025-12-29 14:55:27 UTC) orbiter warning: context deadline exceeded Duration: 5.000898811s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 8.148204376s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:55:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.148452629s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X.XX.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX.] duration: 15.46615979s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:55:05 UTC) orbiter warning: context deadline exceeded Duration: 15.466189126s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....X..............................XXX...........XX.X......] duration: 15.360045151s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:55:04 UTC) orbiter warning: context deadline exceeded Duration: 15.36007688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XXX.XXXXX..XX.X...X.XX..X....XX.XX.X.X.XX...X...X...X....XX] duration: 30.005868132s trace: https://jaeger.dev.storj.io/trace/22589cf4ea1e16a FAIL (2025-12-29 14:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: Q6UKR5ZZMXWLOOKC5HXJ3PDAPPFAGKD7VNXFPK5Y4DYAZRCV5V7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: QTR4MYNL2HZZKOHHT4AOTGSYFEXVPEMAMJWQY7RCLMZPOUUS7CWQ) context deadline exceeded ======LOGS====== Dec 29 14:54:33.713523119 (575h25m33.584606164s) [upload=504441]: Starting upload Dec 29 14:54:33.713605617 (575h25m33.584688682s) [upload=504441]: Done starting upload Dec 29 14:54:33.713777434 (575h25m33.584860479s) [upload=504441]: Uploading segments... Dec 29 14:54:33.713910888 (575h25m33.584993933s) [upload=504441]: Got next segment. Inline: false Dec 29 14:54:33.713953018 (575h25m33.585036063s) [upload=504441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:54:33.713960142 (575h25m33.585043177s) [upload=504441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:54:33.961038255 (575h25m33.83212129s) [seg_pos={0 0}, upload=504441]: Begin upload segment... Dec 29 14:54:33.961104091 (575h25m33.832187136s) [seg_pos={0 0}, upload=504441]: Done begin upload segment. Dec 29 14:54:33.961129138 (575h25m33.832212173s) [noise=true , address=45.91.116.13:4156 , piece_id=KMHJ5MR7B3TJNM5HFCR6M6DKNAHFMMHPBF3PZNGWWWX5HRPL3IBQ, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.961211305 (575h25m33.83229434s) [noise=true , address=45.91.116.14:4191 , piece_id=LUMG7Z55R773OEE5FMEH7LZHGOBV3F4IG47SDX3247XFVWMAK6ZQ, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.961749128 (575h25m33.832832163s) [noise=true , address=45.91.116.4:4940 , piece_id=7YCAGYUBNME75HHDPKBPFBSOMATOZGZSD7DKCJ52FMMGMVDFYBHA, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.962077304 (575h25m33.833160339s) [noise=true , address=145.79.227.17:4972 , piece_id=LX74L7DKQJW5ZHWZ4SJPLYY2X3BVKKMCRFE3Q42TI7GKLNOOGFSA, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.962378697 (575h25m33.833461732s) [noise=true , address=45.91.116.9:4263 , piece_id=Q6UKR5ZZMXWLOOKC5HXJ3PDAPPFAGKD7VNXFPK5Y4DYAZRCV5V7A, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.962668188 (575h25m33.833751223s) [noise=true , address=145.79.227.3:4149 , piece_id=5L4EJO3VIWK7NBCV3GMJGBPF4NZXF5B22AZ46UL7O5YK6IYGTESA, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.962949294 (575h25m33.834032329s) [noise=true , address=145.79.227.18:4007 , piece_id=QFCYSKNSILEEWIRJHRZSX7NEBEYRS5PS3WDKTDSKHE3KRFI7XQQA, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.962959363 (575h25m33.834042398s) [noise=true , address=145.79.227.10:4401 , piece_id=OK472GWUXHN3RZYU3TRHNVVWEAK74PJXZPXC7MTU5A5Z7KDK3AEQ, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.963256669 (575h25m33.834339704s) [noise=true , address=45.91.116.7:4012 , piece_id=QTR4MYNL2HZZKOHHT4AOTGSYFEXVPEMAMJWQY7RCLMZPOUUS7CWQ, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.963302557 (575h25m33.834385592s) [noise=true , address=145.79.227.16:4911 , piece_id=ZCEDXNU3Y3DAE3CE74WEYILNVSY2MSE722FOTURHG5JLWVWBJNBQ, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.963661490 (575h25m33.834744525s) [noise=true , address=145.79.227.39:4014 , piece_id=77CNCFMHIMEU2LWIO42A4TZTHRAZYEACHTHVBA4ZCIHLJPQSYPPA, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.964053616 (575h25m33.835136651s) [noise=true , address=45.91.116.5:4299 , piece_id=PVYQTHHHJPMEYDQBS3V7EFVHG6ABRCREAAMFJK7K35OIHHLR7VAQ, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.964369438 (575h25m33.835452473s) [noise=true , address=145.79.227.40:4049 , piece_id=UFW5Z2HPSUKTGI47I6OMC2QJTPHMR56QLDS3EUP4KDW23BJ63MQQ, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.964914175 (575h25m33.83599721s) [noise=true , address=145.79.227.50:4373 , piece_id=I34IN4T3SNAQGFXOOZIXVSLDXXTB4ZEU37UZKYLUQJTXYMLZS33A, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.965324727 (575h25m33.836407762s) [noise=true , address=145.79.227.46:4603 , piece_id=WS5TNBAUWSSBXQ7DCCVSG7P6ZZYFKM3JMKOVXVZWLTJUCNZNYYWA, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:33.971158375 (575h25m33.84224142s) [noise=true , address=145.79.227.52:4446 , piece_id=H2HOQADTUHZR6KVZVZBOZK5M5H37DJW4T54RY4LSO3REA6L3EJQQ, seg_pos={0 0}, upload=504441]: Uploading piece... Dec 29 14:54:35.679132520 (575h25m35.550215555s) [upload=504441]: Next returned nil segment Dec 29 14:54:35.679145505 (575h25m35.55022855s) [upload=504441]: Waiting for error group managing segments... Dec 29 14:54:35.958204776 (575h25m35.829287811s) [noise=true , address=145.79.227.18:4007 , piece_id=QFCYSKNSILEEWIRJHRZSX7NEBEYRS5PS3WDKTDSKHE3KRFI7XQQA, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:35.962704805 (575h25m35.83378785s) [noise=true , address=145.79.227.16:4911 , piece_id=ZCEDXNU3Y3DAE3CE74WEYILNVSY2MSE722FOTURHG5JLWVWBJNBQ, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:35.973628785 (575h25m35.84471182s) [noise=true , address=145.79.227.10:4401 , piece_id=OK472GWUXHN3RZYU3TRHNVVWEAK74PJXZPXC7MTU5A5Z7KDK3AEQ, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:35.975224874 (575h25m35.846307919s) [noise=true , address=145.79.227.50:4373 , piece_id=I34IN4T3SNAQGFXOOZIXVSLDXXTB4ZEU37UZKYLUQJTXYMLZS33A, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:35.976261058 (575h25m35.847344103s) [noise=true , address=145.79.227.17:4972 , piece_id=LX74L7DKQJW5ZHWZ4SJPLYY2X3BVKKMCRFE3Q42TI7GKLNOOGFSA, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:35.978303637 (575h25m35.849386682s) [noise=true , address=145.79.227.39:4014 , piece_id=77CNCFMHIMEU2LWIO42A4TZTHRAZYEACHTHVBA4ZCIHLJPQSYPPA, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:35.982232518 (575h25m35.853315563s) [noise=true , address=145.79.227.46:4603 , piece_id=WS5TNBAUWSSBXQ7DCCVSG7P6ZZYFKM3JMKOVXVZWLTJUCNZNYYWA, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:35.988488581 (575h25m35.859571616s) [noise=true , address=145.79.227.52:4446 , piece_id=H2HOQADTUHZR6KVZVZBOZK5M5H37DJW4T54RY4LSO3REA6L3EJQQ, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:35.989138798 (575h25m35.860221843s) [noise=true , address=145.79.227.3:4149 , piece_id=5L4EJO3VIWK7NBCV3GMJGBPF4NZXF5B22AZ46UL7O5YK6IYGTESA, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:36.306218174 (575h25m36.177301219s) [noise=true , address=45.91.116.7:4012 , piece_id=QTR4MYNL2HZZKOHHT4AOTGSYFEXVPEMAMJWQY7RCLMZPOUUS7CWQ, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:36.568472004 (575h25m36.439555039s) [noise=true , address=45.91.116.5:4299 , piece_id=PVYQTHHHJPMEYDQBS3V7EFVHG6ABRCREAAMFJK7K35OIHHLR7VAQ, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:36.656134226 (575h25m36.527217261s) [noise=true , address=45.91.116.4:4940 , piece_id=7YCAGYUBNME75HHDPKBPFBSOMATOZGZSD7DKCJ52FMMGMVDFYBHA, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:36.686719335 (575h25m36.55780237s) [noise=true , address=45.91.116.9:4263 , piece_id=Q6UKR5ZZMXWLOOKC5HXJ3PDAPPFAGKD7VNXFPK5Y4DYAZRCV5V7A, seg_pos={0 0}, upload=504441]: Done uploading piece. err: <nil> Dec 29 14:54:36.686737869 (575h25m36.557820904s) [seg_pos={0 0}, upload=504441]: Segment reached optimal threshold of 13 pieces. Dec 29 14:54:36.686912722 (575h25m36.557995757s) [noise=true , address=45.91.116.14:4191 , piece_id=LUMG7Z55R773OEE5FMEH7LZHGOBV3F4IG47SDX3247XFVWMAK6ZQ, seg_pos={0 0}, upload=504441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:50390->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:50390->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50390->45.91.116.14:4191: use of closed network connection Dec 29 14:54:36.686919655 (575h25m36.5580027s) [noise=true , address=45.91.116.14:4191 , piece_id=LUMG7Z55R773OEE5FMEH7LZHGOBV3F4IG47SDX3247XFVWMAK6ZQ, seg_pos={0 0}, upload=504441]: Long tail cancellation due to optimal threshold Dec 29 14:54:36.687017001 (575h25m36.558100056s) [noise=true , address=145.79.227.40:4049 , piece_id=UFW5Z2HPSUKTGI47I6OMC2QJTPHMR56QLDS3EUP4KDW23BJ63MQQ, seg_pos={0 0}, upload=504441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 14:54:36.687049413 (575h25m36.558132448s) [noise=true , address=145.79.227.40:4049 , piece_id=UFW5Z2HPSUKTGI47I6OMC2QJTPHMR56QLDS3EUP4KDW23BJ63MQQ, seg_pos={0 0}, upload=504441]: Long tail cancellation due to optimal threshold Dec 29 14:54:36.687128824 (575h25m36.558211859s) [noise=true , address=45.91.116.13:4156 , piece_id=KMHJ5MR7B3TJNM5HFCR6M6DKNAHFMMHPBF3PZNGWWWX5HRPL3IBQ, seg_pos={0 0}, upload=504441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 14:54:36.687138543 (575h25m36.558221578s) [noise=true , address=45.91.116.13:4156 , piece_id=KMHJ5MR7B3TJNM5HFCR6M6DKNAHFMMHPBF3PZNGWWWX5HRPL3IBQ, seg_pos={0 0}, upload=504441]: Long tail cancellation due to optimal threshold Dec 29 14:54:36.687233523 (575h25m36.558316568s) [seg_pos={0 0}, upload=504441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:54:36.687318485 (575h25m36.55840153s) [upload=504441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:54:36.687324607 (575h25m36.558407652s) [upload=504441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:54:36.984206243 (575h25m36.855289278s) [upload=504441]: Done uploading segments... Dec 29 14:54:36.984225680 (575h25m36.855308715s) [upload=504441]: Upload finished. err: <nil> Duration: 30.005939449s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 5.001600425s trace: https://jaeger.dev.storj.io/trace/4db07ea5c352c278 FAIL (2025-12-29 14:54:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001614662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 7.472962764s trace: https://jaeger.dev.storj.io/trace/6d06116a481c3bb9 FAIL (2025-12-29 14:54:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.472989113s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX.......................X........X.........X......X......] duration: 45.27980038s trace: https://jaeger.dev.storj.io/trace/6478a0d64487577a FAIL (2025-12-29 14:54:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4771 , piece_id=Q6LDYGDGYLATAH7JOZB2OUO2IYB552S4KQXLAEBQBP2IQJB3DEHA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.251070129 (575h19m11.122153164s) [noise=true , address=77.90.31.19:4863 , piece_id=PEP4JSYV5D7FMKQ7C3YKEDQDZRSEPWG5JDSNHI2ZLRADV2UCL57A, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.251437850 (575h19m11.122520885s) [noise=true , address=145.79.227.33:4886 , piece_id=DUAYROBYHSTD7JYTY4NG4ICWZL4BZWYXBJ3REKXE266LEMJSNNMA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.251817152 (575h19m11.122900197s) [noise=true , address=77.90.31.14:4682 , piece_id=CDHWFERWSBUZDKR4CIWQXU542WKXDWXHUSLRJAHQSBUSDOXOVUXA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.252173631 (575h19m11.123256666s) [noise=true , address=145.79.227.29:4965 , piece_id=DDQ5OI3ZTGJW72S7BG4JBNZTFHCEBDUOXWVH4ZXDB6YLXQXTOGRQ, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.252530560 (575h19m11.123613595s) [noise=true , address=64.49.13.170:4723 , piece_id=LFRH7RBK3C5WZSY5BBRXQJBWEMR7E5U4CNN6VH4Q5DLNMXHOZ6OA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.252547924 (575h19m11.123630969s) [noise=true , address=145.79.227.40:4072 , piece_id=DEQR47CTRJ3FGAJZI5WNQOSYZWYYYQ7A76MWD3GKWYH2D62A67WQ, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.252922045 (575h19m11.12400509s) [noise=true , address=77.90.31.18:4838 , piece_id=2EZKOWKFXCIP5XNLQ34ZH4U2TZVNVON3HXMT6LBWS2BOVAZIUM6Q, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.253001848 (575h19m11.124084893s) [noise=true , address=145.79.227.13:4806 , piece_id=NMBGJGVJRCBDXBUQRGY3HIWN2GLR5G442WVVAOMCPVZTNCAHL5XA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.253314153 (575h19m11.124397188s) [noise=true , address=77.90.31.13:4643 , piece_id=XCU3O7UI2ZLWTFUUFL2VYIFNTZYAVXQQZMEEQ27UIRQQUOPDVQSA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.253496950 (575h19m11.124579995s) [noise=true , address=145.79.227.41:4091 , piece_id=NVY2H5JKILJPY7JI35DNVPMECWZARQ4RFYZFONDWGCVM4MVEJAZQ, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.253995289 (575h19m11.125078334s) [noise=true , address=145.79.227.3:4156 , piece_id=XZODCLD2FVVPCGS7NXAJWUADGCRFZHRH4SJLQWIYZTX52R5ENJ7A, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.254466236 (575h19m11.125549281s) [noise=true , address=64.49.13.169:4707 , piece_id=V5M4DJFXHHFJOUYPUEGLZ4SF7GKNU7W3DTSLXZXLVO2WPCXDII3A, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.254911644 (575h19m11.125994689s) [noise=true , address=193.149.164.143:4695 , piece_id=LRP4QCSTS4LM5GR2XCCSDF5FF4JIOZSUVFWMWBQDSUNJLWN4F2KQ, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.255366950 (575h19m11.126449996s) [noise=true , address=45.91.116.13:4173 , piece_id=OSVSBH5XSCDPJRPHRMZBO33FKG67ABSM3Z5TUTV2FPFXK5A7IPSA, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:11.255817408 (575h19m11.126900453s) [noise=true , address=145.79.227.43:4177 , piece_id=AVQC3VFEY63SR5BV4OZI4GHHZXWLNELTOUWVLZNL5V5OFAF7FCSQ, seg_pos={0 0}, upload=504353]: Uploading piece... Dec 29 14:48:12.551291333 (575h19m12.422374368s) [upload=504353]: Next returned nil segment Dec 29 14:48:12.551313084 (575h19m12.422396129s) [upload=504353]: Waiting for error group managing segments... Dec 29 14:48:12.879067587 (575h19m12.750150632s) [noise=true , address=64.49.13.170:4723 , piece_id=LFRH7RBK3C5WZSY5BBRXQJBWEMR7E5U4CNN6VH4Q5DLNMXHOZ6OA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.112148518 (575h19m12.983231563s) [noise=true , address=145.79.227.41:4091 , piece_id=NVY2H5JKILJPY7JI35DNVPMECWZARQ4RFYZFONDWGCVM4MVEJAZQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.112578336 (575h19m12.983661401s) [noise=true , address=145.79.227.13:4806 , piece_id=NMBGJGVJRCBDXBUQRGY3HIWN2GLR5G442WVVAOMCPVZTNCAHL5XA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.122458630 (575h19m12.993541675s) [noise=true , address=145.79.227.29:4965 , piece_id=DDQ5OI3ZTGJW72S7BG4JBNZTFHCEBDUOXWVH4ZXDB6YLXQXTOGRQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.329124643 (575h19m13.200207678s) [noise=true , address=77.90.31.14:4682 , piece_id=CDHWFERWSBUZDKR4CIWQXU542WKXDWXHUSLRJAHQSBUSDOXOVUXA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.336139901 (575h19m13.207222936s) [noise=true , address=77.90.31.13:4643 , piece_id=XCU3O7UI2ZLWTFUUFL2VYIFNTZYAVXQQZMEEQ27UIRQQUOPDVQSA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.427538136 (575h19m13.298621181s) [noise=true , address=77.90.31.18:4838 , piece_id=2EZKOWKFXCIP5XNLQ34ZH4U2TZVNVON3HXMT6LBWS2BOVAZIUM6Q, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.628870834 (575h19m13.499953869s) [noise=true , address=45.91.116.13:4173 , piece_id=OSVSBH5XSCDPJRPHRMZBO33FKG67ABSM3Z5TUTV2FPFXK5A7IPSA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.956600930 (575h19m13.827683965s) [noise=true , address=145.79.227.3:4156 , piece_id=XZODCLD2FVVPCGS7NXAJWUADGCRFZHRH4SJLQWIYZTX52R5ENJ7A, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:13.982179565 (575h19m13.8532626s) [noise=true , address=193.149.164.143:4695 , piece_id=LRP4QCSTS4LM5GR2XCCSDF5FF4JIOZSUVFWMWBQDSUNJLWN4F2KQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:14.019680373 (575h19m13.890763408s) [noise=true , address=145.79.227.34:4936 , piece_id=R5UZN7BR2YFU7JMGIGZUEXSMS6ECMGS66ZQZKY3INVLDSNYHQ4OQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:14.034961690 (575h19m13.906044735s) [noise=true , address=145.79.227.43:4177 , piece_id=AVQC3VFEY63SR5BV4OZI4GHHZXWLNELTOUWVLZNL5V5OFAF7FCSQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:14.090555199 (575h19m13.961638234s) [noise=true , address=145.79.227.44:4210 , piece_id=MCA3TWGDIW76ZSTBFZRCI7D7NSFC6IQ3E7NEG4I3ZR7OY3M7VCIQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:14.130988049 (575h19m14.002071094s) [noise=true , address=145.79.227.30:4771 , piece_id=Q6LDYGDGYLATAH7JOZB2OUO2IYB552S4KQXLAEBQBP2IQJB3DEHA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: <nil> Dec 29 14:48:14.131012024 (575h19m14.002095089s) [seg_pos={0 0}, upload=504353]: Segment reached optimal threshold of 14 pieces. Dec 29 14:48:14.131247373 (575h19m14.002330408s) [noise=true , address=145.79.227.33:4886 , piece_id=DUAYROBYHSTD7JYTY4NG4ICWZL4BZWYXBJ3REKXE266LEMJSNNMA, seg_pos={0 0}, upload=504353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF Dec 29 14:48:14.131256830 (575h19m14.002339895s) [noise=true , address=145.79.227.33:4886 , piece_id=DUAYROBYHSTD7JYTY4NG4ICWZL4BZWYXBJ3REKXE266LEMJSNNMA, seg_pos={0 0}, upload=504353]: Long tail cancellation due to optimal threshold Dec 29 14:48:14.131360277 (575h19m14.002443312s) [noise=true , address=64.49.13.169:4707 , piece_id=V5M4DJFXHHFJOUYPUEGLZ4SF7GKNU7W3DTSLXZXLVO2WPCXDII3A, seg_pos={0 0}, upload=504353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:40890->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:40890->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40890->64.49.13.169:4707: use of closed network connection Dec 29 14:48:14.131381588 (575h19m14.002464653s) [noise=true , address=64.49.13.169:4707 , piece_id=V5M4DJFXHHFJOUYPUEGLZ4SF7GKNU7W3DTSLXZXLVO2WPCXDII3A, seg_pos={0 0}, upload=504353]: Long tail cancellation due to optimal threshold Dec 29 14:48:14.131587190 (575h19m14.002670275s) [noise=true , address=77.90.31.19:4863 , piece_id=PEP4JSYV5D7FMKQ7C3YKEDQDZRSEPWG5JDSNHI2ZLRADV2UCL57A, seg_pos={0 0}, upload=504353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF Dec 29 14:48:14.131596728 (575h19m14.002679773s) [noise=true , address=77.90.31.19:4863 , piece_id=PEP4JSYV5D7FMKQ7C3YKEDQDZRSEPWG5JDSNHI2ZLRADV2UCL57A, seg_pos={0 0}, upload=504353]: Long tail cancellation due to optimal threshold Dec 29 14:48:14.131674396 (575h19m14.002757441s) [noise=true , address=145.79.227.40:4072 , piece_id=DEQR47CTRJ3FGAJZI5WNQOSYZWYYYQ7A76MWD3GKWYH2D62A67WQ, seg_pos={0 0}, upload=504353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 29 14:48:14.131704453 (575h19m14.002787498s) [noise=true , address=145.79.227.40:4072 , piece_id=DEQR47CTRJ3FGAJZI5WNQOSYZWYYYQ7A76MWD3GKWYH2D62A67WQ, seg_pos={0 0}, upload=504353]: Long tail cancellation due to optimal threshold Dec 29 14:48:14.131739099 (575h19m14.002822134s) [seg_pos={0 0}, upload=504353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:48:14.131846874 (575h19m14.002929929s) [upload=504353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:48:14.131873054 (575h19m14.002956089s) [upload=504353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:48:14.446199770 (575h19m14.317282815s) [upload=504353]: Done uploading segments... Dec 29 14:48:14.446218866 (575h19m14.317301901s) [upload=504353]: Upload finished. err: <nil> Duration: 45.279820749s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X..X.XX.XXX.X.X...XX...XX.X.XXXXX.X..X.....X....X....X...XX] duration: 30.001780442s trace: https://jaeger.dev.storj.io/trace/1deb1e9296f0bf75 FAIL (2025-12-29 14:54:07 UTC) orbiter warning: context deadline exceeded Duration: 30.001791674s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [......X..............................XXX...........XX.X.....] duration: 18.274624684s trace: https://jaeger.dev.storj.io/trace/601575448e35e827 FAIL (2025-12-29 14:54:00 UTC) orbiter warning: context deadline exceeded Duration: 18.274653659s core uplink us-select-1 us1-placement-20

1
...
183
184
185
186
187
...
6981