Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXXXX...] duration: 37.68360594s FAIL (2025-08-07 23:18:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 37.683632111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X...X..X.] duration: 29.868620144s FAIL (2025-08-07 23:18:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 29.868639492s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXXXXX.] duration: 59.539580585s FAIL (2025-08-07 23:18:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 59.539594721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................................X..X.XXXX.] duration: 6.715355405s FAIL (2025-08-07 23:18:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== upload=14088]: Done begin upload segment. Aug 7 23:14:58.174494114 (13h34m57.763839101s) [noise=true , address=104.36.82.4:4801 , piece_id=PWV7TZNRT5SV5MDH4F3LVVZYPPY5E53F2QB6D3APCET36JQ7G2WQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174594706 (13h34m57.763939693s) [noise=true , address=77.90.31.17:4771 , piece_id=OBAQRZSSIMSEBMGQ3OVE7BVT5L6BRAST4SU4YEL2HQR3TB4C7RWA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174917945 (13h34m57.764262922s) [noise=true , address=104.36.82.11:4113 , piece_id=ZFACTZDEWA2A2ILT47CFS4DXXVCCUXOJ3FAKN3GCOQW4PBQDGP7A, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174963081 (13h34m57.764308068s) [noise=true , address=77.90.31.13:4627 , piece_id=ZPVXMNQC6C7CDYVTPQO4ZWHRT6H4G643QHZXLSMTY3ROAQDA2PAQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175227337 (13h34m57.764572314s) [noise=true , address=45.91.116.13:4154 , piece_id=LK6FRJB4SOGTCSDGBMO7M2UO532GHS3YSBB3UBQJ5MQQSOKRNBKA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175234811 (13h34m57.764579798s) [noise=true , address=104.36.82.15:4257 , piece_id=QO4HU2TMEQJVVT4IJWQBXFIHWWYC3RNO67BHEH2D3M3COEO5WGHA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175498105 (13h34m57.764843092s) [noise=true , address=77.90.31.18:4806 , piece_id=3ZQ5IIU34KBAMPDBA676VXYJNVCJLDJ2LJEMGQUVUPU4SL55C5TQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175523783 (13h34m57.76486877s) [noise=true , address=45.91.116.11:4117 , piece_id=DPVFTWIVMADRY4KGDQHM7VEMCMZCME56WS3E7D2SJM36XGZ5ERBA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175757932 (13h34m57.765102919s) [noise=true , address=45.91.116.14:4190 , piece_id=DE6K7C7L2SCIWO3OH2DPBUVMECGTDSUUGKZQLQ2PA22J4B4RPR5Q, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175784072 (13h34m57.765129049s) [noise=true , address=104.36.82.9:4006 , piece_id=G5E2Y462PS3JPXHMF2OABO6IJL27KLK34USATI7AY5GV2SSPYKHQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176054940 (13h34m57.765399927s) [noise=true , address=77.90.31.11:4553 , piece_id=2TEUIH35CQDRQY6KMY4KRXHOIV2RRMB5WRTOHW56M7BG7BY4WFZQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176240194 (13h34m57.765585181s) [noise=true , address=45.91.116.5:4297 , piece_id=V7BR6TM3LQM3ERWNGT3O7KYJV4XNVWYGKASOWDG5NPJ6HMSGLAAA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176396734 (13h34m57.765741721s) [noise=true , address=77.90.31.16:4735 , piece_id=ATYURAKL6O2SFDK4ARD5ERDIZGVTJ34F5JBBULRNYETGFRSOUCMQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176542252 (13h34m57.765887239s) [noise=true , address=45.91.116.10:4082 , piece_id=SCGMY34O4OZF4FC63ZSE3KPDK4A2AF55ASQOPQETC73GCCJASCZA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176624790 (13h34m57.765969777s) [noise=true , address=45.91.116.9:4262 , piece_id=7CSV4KG24RRZOTWHO4HBBFCOHBR7WHFSWYI2JQ7D7GQVRO5QPTWQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176827989 (13h34m57.766172966s) [noise=true , address=77.90.31.21:4878 , piece_id=RDAB6QNSRNPWMNFASKEKGHLUWSTS5AQGV5MZU5EW7F7WTX4RBIRA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:59.945982215 (13h34m59.535327192s) [upload=14088]: Next returned nil segment Aug 7 23:14:59.946000040 (13h34m59.535345027s) [upload=14088]: Waiting for error group managing segments... Aug 7 23:15:00.235597016 (13h34m59.824942003s) [noise=true , address=104.36.82.4:4801 , piece_id=PWV7TZNRT5SV5MDH4F3LVVZYPPY5E53F2QB6D3APCET36JQ7G2WQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.336486456 (13h34m59.925831443s) [noise=true , address=104.36.82.9:4006 , piece_id=G5E2Y462PS3JPXHMF2OABO6IJL27KLK34USATI7AY5GV2SSPYKHQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.461953790 (13h35m0.051298777s) [noise=true , address=104.36.82.15:4257 , piece_id=QO4HU2TMEQJVVT4IJWQBXFIHWWYC3RNO67BHEH2D3M3COEO5WGHA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.536493809 (13h35m0.125838786s) [noise=true , address=77.90.31.17:4771 , piece_id=OBAQRZSSIMSEBMGQ3OVE7BVT5L6BRAST4SU4YEL2HQR3TB4C7RWA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.547607852 (13h35m0.13695284s) [noise=true , address=77.90.31.16:4735 , piece_id=ATYURAKL6O2SFDK4ARD5ERDIZGVTJ34F5JBBULRNYETGFRSOUCMQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.557092168 (13h35m0.146437155s) [noise=true , address=45.91.116.9:4262 , piece_id=7CSV4KG24RRZOTWHO4HBBFCOHBR7WHFSWYI2JQ7D7GQVRO5QPTWQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.588663019 (13h35m0.178007996s) [noise=true , address=77.90.31.11:4553 , piece_id=2TEUIH35CQDRQY6KMY4KRXHOIV2RRMB5WRTOHW56M7BG7BY4WFZQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.669587042 (13h35m0.258932019s) [noise=true , address=104.36.82.11:4113 , piece_id=ZFACTZDEWA2A2ILT47CFS4DXXVCCUXOJ3FAKN3GCOQW4PBQDGP7A, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.772791433 (13h35m0.36213642s) [noise=true , address=45.91.116.11:4117 , piece_id=DPVFTWIVMADRY4KGDQHM7VEMCMZCME56WS3E7D2SJM36XGZ5ERBA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.788386820 (13h35m0.377731797s) [noise=true , address=77.90.31.13:4627 , piece_id=ZPVXMNQC6C7CDYVTPQO4ZWHRT6H4G643QHZXLSMTY3ROAQDA2PAQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.856809245 (13h35m0.446154232s) [noise=true , address=45.91.116.10:4082 , piece_id=SCGMY34O4OZF4FC63ZSE3KPDK4A2AF55ASQOPQETC73GCCJASCZA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.865234375 (13h35m0.454579362s) [noise=true , address=45.91.116.14:4190 , piece_id=DE6K7C7L2SCIWO3OH2DPBUVMECGTDSUUGKZQLQ2PA22J4B4RPR5Q, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.886019812 (13h35m0.475364799s) [noise=true , address=77.90.31.21:4878 , piece_id=RDAB6QNSRNPWMNFASKEKGHLUWSTS5AQGV5MZU5EW7F7WTX4RBIRA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.886047264 (13h35m0.475392241s) [seg_pos={0 0}, upload=14088]: Segment reached optimal threshold of 13 pieces. Aug 7 23:15:00.886165169 (13h35m0.475510156s) [noise=true , address=45.91.116.13:4154 , piece_id=LK6FRJB4SOGTCSDGBMO7M2UO532GHS3YSBB3UBQJ5MQQSOKRNBKA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886224553 (13h35m0.47556954s) [noise=true , address=45.91.116.5:4297 , piece_id=V7BR6TM3LQM3ERWNGT3O7KYJV4XNVWYGKASOWDG5NPJ6HMSGLAAA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886291521 (13h35m0.475636498s) [noise=true , address=77.90.31.18:4806 , piece_id=3ZQ5IIU34KBAMPDBA676VXYJNVCJLDJ2LJEMGQUVUPU4SL55C5TQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886340935 (13h35m0.475685922s) [seg_pos={0 0}, upload=14088]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:15:00.886432311 (13h35m0.475777298s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:15:00.886436909 (13h35m0.475781896s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:49.547592064 (13h36m49.136937041s) [upload=14088]: Done uploading segments... Aug 7 23:16:49.547614097 (13h36m49.136959084s) [upload=14088]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:50.417681936 (13h36m50.007026923s) [upload=14116]: Starting upload Aug 7 23:16:50.417744316 (13h36m50.007089294s) [upload=14116]: Done starting upload Aug 7 23:16:50.417844638 (13h36m50.007189625s) [upload=14116]: Uploading segments... Aug 7 23:16:50.417958475 (13h36m50.007303462s) [upload=14116]: Got next segment. Inline: false Aug 7 23:16:50.417989144 (13h36m50.007334131s) [upload=14116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:50.417993503 (13h36m50.00733849s) [upload=14116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:17:19.777322561 (13h37m19.366667548s) [upload=14116]: Next segment err: context deadline exceeded Aug 7 23:17:19.777345936 (13h37m19.366690923s) [upload=14116]: Done uploading segments... Aug 7 23:17:19.777363158 (13h37m19.366708166s) [upload=14116]: Upload finished. err: context deadline exceeded Aug 7 23:17:33.537572296 (13h37m33.126917283s) [upload=14128]: Starting upload Aug 7 23:17:33.537636568 (13h37m33.126981555s) [upload=14128]: Done starting upload Aug 7 23:17:33.537721962 (13h37m33.127066949s) [upload=14128]: Uploading segments... Aug 7 23:17:33.538074877 (13h37m33.127419904s) [upload=14128]: Got next segment. Inline: false Aug 7 23:17:33.538146895 (13h37m33.127491932s) [upload=14128]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:33.538156122 (13h37m33.12750114s) [upload=14128]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:02.878943122 (13h38m2.468288109s) [upload=14128]: Next segment err: context deadline exceeded Aug 7 23:18:02.879172691 (13h38m2.468517718s) [upload=14128]: Done uploading segments... Aug 7 23:18:02.879209642 (13h38m2.468554679s) [upload=14128]: Upload finished. err: context deadline exceeded Duration: 6.715382777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXXX.] duration: 1.897318442s FAIL (2025-08-07 23:18:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.89735432s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.00052478s FAIL (2025-08-07 23:18:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000546151s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 655.408199ms target: https://us1.storj.io/login FAIL (2025-08-07 23:18:25 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 655.440801ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 641.713656ms target: https://us1.storj.io/login FAIL (2025-08-07 23:18:04 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 641.73744ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................................X..X.XXXX] duration: 30.000831082s FAIL (2025-08-07 23:18:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== upload=14088]: Done begin upload segment. Aug 7 23:14:58.174494114 (13h34m57.763839101s) [noise=true , address=104.36.82.4:4801 , piece_id=PWV7TZNRT5SV5MDH4F3LVVZYPPY5E53F2QB6D3APCET36JQ7G2WQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174594706 (13h34m57.763939693s) [noise=true , address=77.90.31.17:4771 , piece_id=OBAQRZSSIMSEBMGQ3OVE7BVT5L6BRAST4SU4YEL2HQR3TB4C7RWA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174917945 (13h34m57.764262922s) [noise=true , address=104.36.82.11:4113 , piece_id=ZFACTZDEWA2A2ILT47CFS4DXXVCCUXOJ3FAKN3GCOQW4PBQDGP7A, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174963081 (13h34m57.764308068s) [noise=true , address=77.90.31.13:4627 , piece_id=ZPVXMNQC6C7CDYVTPQO4ZWHRT6H4G643QHZXLSMTY3ROAQDA2PAQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175227337 (13h34m57.764572314s) [noise=true , address=45.91.116.13:4154 , piece_id=LK6FRJB4SOGTCSDGBMO7M2UO532GHS3YSBB3UBQJ5MQQSOKRNBKA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175234811 (13h34m57.764579798s) [noise=true , address=104.36.82.15:4257 , piece_id=QO4HU2TMEQJVVT4IJWQBXFIHWWYC3RNO67BHEH2D3M3COEO5WGHA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175498105 (13h34m57.764843092s) [noise=true , address=77.90.31.18:4806 , piece_id=3ZQ5IIU34KBAMPDBA676VXYJNVCJLDJ2LJEMGQUVUPU4SL55C5TQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175523783 (13h34m57.76486877s) [noise=true , address=45.91.116.11:4117 , piece_id=DPVFTWIVMADRY4KGDQHM7VEMCMZCME56WS3E7D2SJM36XGZ5ERBA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175757932 (13h34m57.765102919s) [noise=true , address=45.91.116.14:4190 , piece_id=DE6K7C7L2SCIWO3OH2DPBUVMECGTDSUUGKZQLQ2PA22J4B4RPR5Q, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175784072 (13h34m57.765129049s) [noise=true , address=104.36.82.9:4006 , piece_id=G5E2Y462PS3JPXHMF2OABO6IJL27KLK34USATI7AY5GV2SSPYKHQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176054940 (13h34m57.765399927s) [noise=true , address=77.90.31.11:4553 , piece_id=2TEUIH35CQDRQY6KMY4KRXHOIV2RRMB5WRTOHW56M7BG7BY4WFZQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176240194 (13h34m57.765585181s) [noise=true , address=45.91.116.5:4297 , piece_id=V7BR6TM3LQM3ERWNGT3O7KYJV4XNVWYGKASOWDG5NPJ6HMSGLAAA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176396734 (13h34m57.765741721s) [noise=true , address=77.90.31.16:4735 , piece_id=ATYURAKL6O2SFDK4ARD5ERDIZGVTJ34F5JBBULRNYETGFRSOUCMQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176542252 (13h34m57.765887239s) [noise=true , address=45.91.116.10:4082 , piece_id=SCGMY34O4OZF4FC63ZSE3KPDK4A2AF55ASQOPQETC73GCCJASCZA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176624790 (13h34m57.765969777s) [noise=true , address=45.91.116.9:4262 , piece_id=7CSV4KG24RRZOTWHO4HBBFCOHBR7WHFSWYI2JQ7D7GQVRO5QPTWQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176827989 (13h34m57.766172966s) [noise=true , address=77.90.31.21:4878 , piece_id=RDAB6QNSRNPWMNFASKEKGHLUWSTS5AQGV5MZU5EW7F7WTX4RBIRA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:59.945982215 (13h34m59.535327192s) [upload=14088]: Next returned nil segment Aug 7 23:14:59.946000040 (13h34m59.535345027s) [upload=14088]: Waiting for error group managing segments... Aug 7 23:15:00.235597016 (13h34m59.824942003s) [noise=true , address=104.36.82.4:4801 , piece_id=PWV7TZNRT5SV5MDH4F3LVVZYPPY5E53F2QB6D3APCET36JQ7G2WQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.336486456 (13h34m59.925831443s) [noise=true , address=104.36.82.9:4006 , piece_id=G5E2Y462PS3JPXHMF2OABO6IJL27KLK34USATI7AY5GV2SSPYKHQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.461953790 (13h35m0.051298777s) [noise=true , address=104.36.82.15:4257 , piece_id=QO4HU2TMEQJVVT4IJWQBXFIHWWYC3RNO67BHEH2D3M3COEO5WGHA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.536493809 (13h35m0.125838786s) [noise=true , address=77.90.31.17:4771 , piece_id=OBAQRZSSIMSEBMGQ3OVE7BVT5L6BRAST4SU4YEL2HQR3TB4C7RWA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.547607852 (13h35m0.13695284s) [noise=true , address=77.90.31.16:4735 , piece_id=ATYURAKL6O2SFDK4ARD5ERDIZGVTJ34F5JBBULRNYETGFRSOUCMQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.557092168 (13h35m0.146437155s) [noise=true , address=45.91.116.9:4262 , piece_id=7CSV4KG24RRZOTWHO4HBBFCOHBR7WHFSWYI2JQ7D7GQVRO5QPTWQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.588663019 (13h35m0.178007996s) [noise=true , address=77.90.31.11:4553 , piece_id=2TEUIH35CQDRQY6KMY4KRXHOIV2RRMB5WRTOHW56M7BG7BY4WFZQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.669587042 (13h35m0.258932019s) [noise=true , address=104.36.82.11:4113 , piece_id=ZFACTZDEWA2A2ILT47CFS4DXXVCCUXOJ3FAKN3GCOQW4PBQDGP7A, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.772791433 (13h35m0.36213642s) [noise=true , address=45.91.116.11:4117 , piece_id=DPVFTWIVMADRY4KGDQHM7VEMCMZCME56WS3E7D2SJM36XGZ5ERBA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.788386820 (13h35m0.377731797s) [noise=true , address=77.90.31.13:4627 , piece_id=ZPVXMNQC6C7CDYVTPQO4ZWHRT6H4G643QHZXLSMTY3ROAQDA2PAQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.856809245 (13h35m0.446154232s) [noise=true , address=45.91.116.10:4082 , piece_id=SCGMY34O4OZF4FC63ZSE3KPDK4A2AF55ASQOPQETC73GCCJASCZA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.865234375 (13h35m0.454579362s) [noise=true , address=45.91.116.14:4190 , piece_id=DE6K7C7L2SCIWO3OH2DPBUVMECGTDSUUGKZQLQ2PA22J4B4RPR5Q, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.886019812 (13h35m0.475364799s) [noise=true , address=77.90.31.21:4878 , piece_id=RDAB6QNSRNPWMNFASKEKGHLUWSTS5AQGV5MZU5EW7F7WTX4RBIRA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.886047264 (13h35m0.475392241s) [seg_pos={0 0}, upload=14088]: Segment reached optimal threshold of 13 pieces. Aug 7 23:15:00.886165169 (13h35m0.475510156s) [noise=true , address=45.91.116.13:4154 , piece_id=LK6FRJB4SOGTCSDGBMO7M2UO532GHS3YSBB3UBQJ5MQQSOKRNBKA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886224553 (13h35m0.47556954s) [noise=true , address=45.91.116.5:4297 , piece_id=V7BR6TM3LQM3ERWNGT3O7KYJV4XNVWYGKASOWDG5NPJ6HMSGLAAA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886291521 (13h35m0.475636498s) [noise=true , address=77.90.31.18:4806 , piece_id=3ZQ5IIU34KBAMPDBA676VXYJNVCJLDJ2LJEMGQUVUPU4SL55C5TQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886340935 (13h35m0.475685922s) [seg_pos={0 0}, upload=14088]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:15:00.886432311 (13h35m0.475777298s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:15:00.886436909 (13h35m0.475781896s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:49.547592064 (13h36m49.136937041s) [upload=14088]: Done uploading segments... Aug 7 23:16:49.547614097 (13h36m49.136959084s) [upload=14088]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:50.417681936 (13h36m50.007026923s) [upload=14116]: Starting upload Aug 7 23:16:50.417744316 (13h36m50.007089294s) [upload=14116]: Done starting upload Aug 7 23:16:50.417844638 (13h36m50.007189625s) [upload=14116]: Uploading segments... Aug 7 23:16:50.417958475 (13h36m50.007303462s) [upload=14116]: Got next segment. Inline: false Aug 7 23:16:50.417989144 (13h36m50.007334131s) [upload=14116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:50.417993503 (13h36m50.00733849s) [upload=14116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:17:19.777322561 (13h37m19.366667548s) [upload=14116]: Next segment err: context deadline exceeded Aug 7 23:17:19.777345936 (13h37m19.366690923s) [upload=14116]: Done uploading segments... Aug 7 23:17:19.777363158 (13h37m19.366708166s) [upload=14116]: Upload finished. err: context deadline exceeded Aug 7 23:17:33.537572296 (13h37m33.126917283s) [upload=14128]: Starting upload Aug 7 23:17:33.537636568 (13h37m33.126981555s) [upload=14128]: Done starting upload Aug 7 23:17:33.537721962 (13h37m33.127066949s) [upload=14128]: Uploading segments... Aug 7 23:17:33.538074877 (13h37m33.127419904s) [upload=14128]: Got next segment. Inline: false Aug 7 23:17:33.538146895 (13h37m33.127491932s) [upload=14128]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:33.538156122 (13h37m33.12750114s) [upload=14128]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:02.878943122 (13h38m2.468288109s) [upload=14128]: Next segment err: context deadline exceeded Aug 7 23:18:02.879172691 (13h38m2.468517718s) [upload=14128]: Done uploading segments... Aug 7 23:18:02.879209642 (13h38m2.468554679s) [upload=14128]: Upload finished. err: context deadline exceeded Duration: 30.000854127s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X.....................................................XXXXXX] duration: 1m0.001327616s FAIL (2025-08-07 23:17:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001339648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X......X] duration: 924.972993ms FAIL (2025-08-07 23:17:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Aug 7 23:13:47.522489627 (13h33m47.111834614s) [upload=14072]: Starting upload Aug 7 23:13:47.522559681 (13h33m47.111904668s) [upload=14072]: Done starting upload Aug 7 23:13:47.522629674 (13h33m47.111974661s) [upload=14072]: Uploading segments... Aug 7 23:13:47.522726771 (13h33m47.112071778s) [upload=14072]: Got next segment. Inline: true Aug 7 23:13:47.522735617 (13h33m47.112080614s) [upload=14072]: Waiting for error group managing segments... Aug 7 23:13:47.522790582 (13h33m47.112135569s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:47.522796493 (13h33m47.112141481s) [upload=14072]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:47.522799320 (13h33m47.112144307s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.796144342 (13h33m47.385489329s) [upload=14072]: Done uploading segments... Aug 7 23:13:47.796162656 (13h33m47.385507643s) [upload=14072]: Upload finished. err: <nil> Aug 7 23:14:22.239921861 (13h34m21.829266848s) [upload=14081]: Starting upload Aug 7 23:14:22.239973791 (13h34m21.829318778s) [upload=14081]: Done starting upload Aug 7 23:14:22.240087399 (13h34m21.829432396s) [upload=14081]: Uploading segments... Aug 7 23:14:22.240187159 (13h34m21.829532156s) [upload=14081]: Got next segment. Inline: true Aug 7 23:14:22.240194182 (13h34m21.829539179s) [upload=14081]: Waiting for error group managing segments... Aug 7 23:14:22.240255971 (13h34m21.829600958s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:22.240261311 (13h34m21.829606288s) [upload=14081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:22.240263816 (13h34m21.829608803s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:22.537482123 (13h34m22.12682711s) [upload=14081]: Done uploading segments... Aug 7 23:14:22.537500677 (13h34m22.126845664s) [upload=14081]: Upload finished. err: <nil> Aug 7 23:14:56.968922831 (13h34m56.558267818s) [upload=14090]: Starting upload Aug 7 23:14:56.968974069 (13h34m56.558319056s) [upload=14090]: Done starting upload Aug 7 23:14:56.969028383 (13h34m56.55837337s) [upload=14090]: Uploading segments... Aug 7 23:14:56.969059602 (13h34m56.55840459s) [upload=14090]: Got next segment. Inline: true Aug 7 23:14:56.969063500 (13h34m56.558408487s) [upload=14090]: Waiting for error group managing segments... Aug 7 23:14:56.969102906 (13h34m56.558447893s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.969107034 (13h34m56.558452021s) [upload=14090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:56.969109899 (13h34m56.558454886s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:57.268863065 (13h34m56.858208052s) [upload=14090]: Done uploading segments... Aug 7 23:14:57.268882201 (13h34m56.858227188s) [upload=14090]: Upload finished. err: <nil> Aug 7 23:16:09.138990771 (13h36m8.728335758s) [upload=14107]: Starting upload Aug 7 23:16:09.139048231 (13h36m8.728393208s) [upload=14107]: Done starting upload Aug 7 23:16:09.139094229 (13h36m8.728439226s) [upload=14107]: Uploading segments... Aug 7 23:16:09.139131069 (13h36m8.728476056s) [upload=14107]: Got next segment. Inline: true Aug 7 23:16:09.139135007 (13h36m8.728479994s) [upload=14107]: Waiting for error group managing segments... Aug 7 23:16:09.139167859 (13h36m8.728512846s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.139171937 (13h36m8.728516924s) [upload=14107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:09.139199730 (13h36m8.728544707s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:09.409287150 (13h36m8.998632137s) [upload=14107]: Done uploading segments... Aug 7 23:16:09.409303952 (13h36m8.998648939s) [upload=14107]: Upload finished. err: <nil> Aug 7 23:16:30.665596455 (13h36m30.254941442s) [upload=14111]: Starting upload Aug 7 23:16:30.665642382 (13h36m30.254987369s) [upload=14111]: Done starting upload Aug 7 23:16:30.665687749 (13h36m30.255032726s) [upload=14111]: Uploading segments... Aug 7 23:16:30.665725801 (13h36m30.255070788s) [upload=14111]: Got next segment. Inline: true Aug 7 23:16:30.665729409 (13h36m30.255074386s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:16:30.665763554 (13h36m30.255108541s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.665768984 (13h36m30.255113971s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.665771289 (13h36m30.255116276s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.945395745 (13h36m30.534740732s) [upload=14111]: Done uploading segments... Aug 7 23:16:30.945418538 (13h36m30.534763526s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:17:25.677744155 (13h37m25.267089143s) [upload=14125]: Starting upload Aug 7 23:17:25.677793930 (13h37m25.267138917s) [upload=14125]: Done starting upload Aug 7 23:17:25.677862562 (13h37m25.267207539s) [upload=14125]: Uploading segments... Aug 7 23:17:25.677896967 (13h37m25.267241954s) [upload=14125]: Got next segment. Inline: true Aug 7 23:17:25.677900704 (13h37m25.267245681s) [upload=14125]: Waiting for error group managing segments... Aug 7 23:17:25.677933046 (13h37m25.267278033s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:25.677936884 (13h37m25.267281871s) [upload=14125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:25.677939309 (13h37m25.267284296s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.502642065 (13h37m26.091987052s) [upload=14125]: Done uploading segments... Aug 7 23:17:26.502660942 (13h37m26.092005919s) [upload=14125]: Upload finished. err: <nil> Aug 7 23:17:53.025114325 (13h37m52.614459313s) [upload=14131]: Starting upload Aug 7 23:17:53.025169641 (13h37m52.614514618s) [upload=14131]: Done starting upload Aug 7 23:17:53.025219857 (13h37m52.614564844s) [upload=14131]: Uploading segments... Aug 7 23:17:53.025267939 (13h37m52.614612936s) [upload=14131]: Got next segment. Inline: true Aug 7 23:17:53.025271916 (13h37m52.614616903s) [upload=14131]: Waiting for error group managing segments... Aug 7 23:17:53.025302495 (13h37m52.614647482s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:53.025306382 (13h37m52.614651369s) [upload=14131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:53.025308497 (13h37m52.614653484s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.302835519 (13h37m52.892180506s) [upload=14131]: Done uploading segments... Aug 7 23:17:53.302854656 (13h37m52.892199633s) [upload=14131]: Upload finished. err: metaclient: Too Many Requests Duration: 924.994755ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X...X..X] duration: 30.001047682s FAIL (2025-08-07 23:17:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001072761s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X...................X....................X.....] duration: 9.124566257s FAIL (2025-08-07 23:17:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e85db788dbeec38ad8084daa355e1bf59b8b718af998fc40c7451ca1ba6627b2 != before 1cad814b0341363008cdb7d22dd66bb6d87f0105cd39850888ced4405a0b4b43 ======LOGS====== upload Aug 7 23:13:43.967401320 (13h33m43.556746307s) [upload=14069]: Uploading segments... Aug 7 23:13:43.967498206 (13h33m43.556843193s) [upload=14069]: Got next segment. Inline: false Aug 7 23:13:43.967560455 (13h33m43.556905442s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:43.967568550 (13h33m43.556913577s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:44.414536988 (13h33m44.003881975s) [seg_pos={0 0}, upload=14069]: Begin upload segment... Aug 7 23:13:44.414587104 (13h33m44.003932091s) [seg_pos={0 0}, upload=14069]: Done begin upload segment. Aug 7 23:13:44.414606933 (13h33m44.00395192s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414666236 (13h33m44.004011223s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414966160 (13h33m44.004311147s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415118381 (13h33m44.004463369s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415228682 (13h33m44.00457367s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415496494 (13h33m44.004841471s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415745370 (13h33m44.005090357s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416026788 (13h33m44.005371766s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416256569 (13h33m44.005601556s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416456080 (13h33m44.005801067s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416552746 (13h33m44.005897753s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416803596 (13h33m44.006148583s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417042221 (13h33m44.006387208s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417277101 (13h33m44.006622088s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417316085 (13h33m44.006661072s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417537459 (13h33m44.006882436s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417738975 (13h33m44.007083952s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417952763 (13h33m44.00729774s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:45.514309387 (13h33m45.103654374s) [upload=14069]: Next returned nil segment Aug 7 23:13:45.514327020 (13h33m45.103672007s) [upload=14069]: Waiting for error group managing segments... Aug 7 23:13:45.795258156 (13h33m45.384603143s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.802263469 (13h33m45.391608456s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.887877424 (13h33m45.477222401s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.350209185 (13h33m45.939554172s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.389672887 (13h33m45.979017874s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.399409194 (13h33m45.988754171s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.424971119 (13h33m46.014316106s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.450918913 (13h33m46.0402639s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.492609174 (13h33m46.081954161s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.516716376 (13h33m46.106061353s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.592315332 (13h33m46.181660319s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.746385819 (13h33m46.335730806s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.786539771 (13h33m46.375884758s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838001699 (13h33m46.427346676s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838017648 (13h33m46.427362636s) [seg_pos={0 0}, upload=14069]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:46.838217681 (13h33m46.427562658s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838422624 (13h33m46.427767601s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838575365 (13h33m46.427920352s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: rpc: dial tcp 104.36.82.16:4296: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838627195 (13h33m46.427972172s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:48060->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838635551 (13h33m46.427980538s) [seg_pos={0 0}, upload=14069]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:46.838702128 (13h33m46.428047115s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:46.838705895 (13h33m46.428050882s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.127771955 (13h33m46.717116932s) [upload=14069]: Done uploading segments... Aug 7 23:13:47.127791082 (13h33m46.717136069s) [upload=14069]: Upload finished. err: <nil> Duration: 9.124591846s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXX.X.] duration: 7.131676615s FAIL (2025-08-07 23:17:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.131698287s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.X....................................................XX.] duration: 15.688831602s FAIL (2025-08-07 23:17:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad=14060]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:09.091387704 (13h33m8.680732691s) [seg_pos={0 0}, upload=14060]: Begin upload segment... Aug 7 23:13:09.091429043 (13h33m8.68077403s) [seg_pos={0 0}, upload=14060]: Done begin upload segment. Aug 7 23:13:09.091448139 (13h33m8.680793127s) [noise=true , address=104.36.82.35:4583 , piece_id=YCF4I7CDHUNPFY3GO5RJSSSELYU4GZFJOBCEIAUOZA5HGVV2AZIQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.091805804 (13h33m8.681150791s) [noise=true , address=104.36.82.5:4843 , piece_id=RFDJLWO26354TA2DOODQ5NWXQGBUCWSBVWNBZLNNHPBUMOQ5YG6A, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.092116358 (13h33m8.681461345s) [noise=true , address=104.36.82.59:4314 , piece_id=W6EMX4SZAY3KXXPV7FJNNZV6UXK2CMUPKB2S4E3PCT75R3P7CJBA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.092420690 (13h33m8.681765667s) [noise=true , address=104.36.82.16:4312 , piece_id=UD46N4FCM7UGXJWUIKG7DYINDDIO5L6KHBLTFXFTYNUYOFJTAOUA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.092787793 (13h33m8.68213278s) [noise=true , address=64.49.13.27:4964 , piece_id=3C44RWLK3JI7BW7II6WHJLY3KIK3FC3U4ZJHL47DJIKN3PVXDSFA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.093363383 (13h33m8.68270837s) [noise=true , address=45.91.116.8:4051 , piece_id=UOEY4RXDEGWOXOS3XHEZ2EZMAFJXNTQFO6ESPDXS3SP4EE4PTOTA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.093583274 (13h33m8.682928261s) [noise=true , address=104.36.82.5:4866 , piece_id=XGJLHUY6KQX7JNJFYR2I7FX4FQGOAIXEFYOZISR246CR3UAKZWYA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.093789509 (13h33m8.683134496s) [noise=true , address=104.36.82.52:4095 , piece_id=72VSSUO6C6ZJ6RJ627ZL6WNWBPKUGTPBWI6GO44ZJFDQYEFRLYLQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.094036702 (13h33m8.683381689s) [noise=true , address=104.36.82.45:4932 , piece_id=RDVHKOUB3MFQXCWQYIP2GTYGLE4A5JORDQO54WWPFERQG7W22ZLQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.094306238 (13h33m8.683651225s) [noise=true , address=79.127.201.215:4347 , piece_id=ICE4FM3Z2PFPLG3VQEXADCCVALDUFBX7RBAAABIBGSSA3BDJBABQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.094581394 (13h33m8.683926381s) [noise=true , address=64.49.13.169:4713 , piece_id=OC4GKJQXE7JXL6J4YNCJERNCDGFA25FDPZ3EOMUJ6JWCXJUTZZXA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.094890105 (13h33m8.684235082s) [noise=true , address=64.49.13.137:4493 , piece_id=WC4R5BUB46HUXV3XKARPMEWWWLXJV5LJ6QNLXIOEWCZBXJYYA6BQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.095243872 (13h33m8.684588849s) [noise=true , address=104.36.82.39:4749 , piece_id=4SYI2SFT6YSMDTOTXJTINJ7LU2LD4ZHAOME4ZZXVKU5CGBNPG6BQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.095576859 (13h33m8.684921846s) [noise=true , address=104.36.82.19:4408 , piece_id=HQQTSW7FJVUAZPVJ5DXFK35RN2LVQAIFOVG3NHVLFL3WRL7AFPHQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.095852887 (13h33m8.685197874s) [noise=true , address=45.91.116.11:4151 , piece_id=KSXLHRUPUUUK4UVIF76ASVMQEUTSIXMNTB5WMZ52NRAE2ZF6M5BQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.096045065 (13h33m8.685390052s) [noise=true , address=104.36.82.43:4863 , piece_id=7ETQFK4AWX2OQOPCQ32OH6VJUAS46GTYXYMNM2Y5JAGRD7NOSMWQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.096498421 (13h33m8.685843409s) [noise=true , address=104.36.82.44:4901 , piece_id=5GNUWDIXFSJLG4QJW7IS5MD2W35YMQMGRYZ5G6JWP2XL43AG76EA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.096818584 (13h33m8.686163571s) [noise=true , address=104.36.82.36:4646 , piece_id=X374LZM3CKNA3PNJ5F4W6KMUQELJWKWAPWXHPNBBPQC4DYJVLWWQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:10.198767377 (13h33m9.788112364s) [upload=14060]: Next returned nil segment Aug 7 23:13:10.198784199 (13h33m9.788129186s) [upload=14060]: Waiting for error group managing segments... Aug 7 23:13:10.482190972 (13h33m10.071535949s) [noise=true , address=64.49.13.137:4493 , piece_id=WC4R5BUB46HUXV3XKARPMEWWWLXJV5LJ6QNLXIOEWCZBXJYYA6BQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:10.513415901 (13h33m10.102760888s) [noise=true , address=64.49.13.27:4964 , piece_id=3C44RWLK3JI7BW7II6WHJLY3KIK3FC3U4ZJHL47DJIKN3PVXDSFA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:10.515976149 (13h33m10.105321136s) [noise=true , address=64.49.13.169:4713 , piece_id=OC4GKJQXE7JXL6J4YNCJERNCDGFA25FDPZ3EOMUJ6JWCXJUTZZXA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.003898079 (13h33m10.593243066s) [noise=true , address=104.36.82.39:4749 , piece_id=4SYI2SFT6YSMDTOTXJTINJ7LU2LD4ZHAOME4ZZXVKU5CGBNPG6BQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.038920043 (13h33m10.62826503s) [noise=true , address=79.127.201.215:4347 , piece_id=ICE4FM3Z2PFPLG3VQEXADCCVALDUFBX7RBAAABIBGSSA3BDJBABQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.061211160 (13h33m10.650556157s) [noise=true , address=104.36.82.43:4863 , piece_id=7ETQFK4AWX2OQOPCQ32OH6VJUAS46GTYXYMNM2Y5JAGRD7NOSMWQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.116268806 (13h33m10.705613793s) [noise=true , address=104.36.82.19:4408 , piece_id=HQQTSW7FJVUAZPVJ5DXFK35RN2LVQAIFOVG3NHVLFL3WRL7AFPHQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.119226344 (13h33m10.708571381s) [noise=true , address=104.36.82.45:4932 , piece_id=RDVHKOUB3MFQXCWQYIP2GTYGLE4A5JORDQO54WWPFERQG7W22ZLQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.163437749 (13h33m10.752782736s) [noise=true , address=104.36.82.44:4901 , piece_id=5GNUWDIXFSJLG4QJW7IS5MD2W35YMQMGRYZ5G6JWP2XL43AG76EA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.166785074 (13h33m10.756130051s) [noise=true , address=104.36.82.36:4646 , piece_id=X374LZM3CKNA3PNJ5F4W6KMUQELJWKWAPWXHPNBBPQC4DYJVLWWQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.190978580 (13h33m10.780323557s) [noise=true , address=104.36.82.35:4583 , piece_id=YCF4I7CDHUNPFY3GO5RJSSSELYU4GZFJOBCEIAUOZA5HGVV2AZIQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.229816815 (13h33m10.819161802s) [noise=true , address=45.91.116.8:4051 , piece_id=UOEY4RXDEGWOXOS3XHEZ2EZMAFJXNTQFO6ESPDXS3SP4EE4PTOTA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.257926594 (13h33m10.847271581s) [noise=true , address=45.91.116.11:4151 , piece_id=KSXLHRUPUUUK4UVIF76ASVMQEUTSIXMNTB5WMZ52NRAE2ZF6M5BQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.261359843 (13h33m10.85070483s) [noise=true , address=104.36.82.52:4095 , piece_id=72VSSUO6C6ZJ6RJ627ZL6WNWBPKUGTPBWI6GO44ZJFDQYEFRLYLQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.261376014 (13h33m10.850721002s) [seg_pos={0 0}, upload=14060]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:11.261727868 (13h33m10.851072855s) [noise=true , address=104.36.82.5:4843 , piece_id=RFDJLWO26354TA2DOODQ5NWXQGBUCWSBVWNBZLNNHPBUMOQ5YG6A, seg_pos={0 0}, upload=14060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HvL2JDP4GCsu3tziVEKRMh14nEzyRMRuWNSmVvs27LzHTbNpS): protocol: write tcp 188.166.225.170:37258->104.36.82.5:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:11.261771851 (13h33m10.851116838s) [noise=true , address=104.36.82.59:4314 , piece_id=W6EMX4SZAY3KXXPV7FJNNZV6UXK2CMUPKB2S4E3PCT75R3P7CJBA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EsmL66Gn6K9cUxHqqGtuUZxUGyWCNL4oqyHzS1XUbWVnMMJbd): protocol: write tcp 188.166.225.170:54566->104.36.82.59:4314: use of closed network connection; write tcp 188.166.225.170:54566->104.36.82.59:4314: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54566->104.36.82.59:4314: use of closed network connection Aug 7 23:13:11.261778304 (13h33m10.851123291s) [noise=true , address=104.36.82.16:4312 , piece_id=UD46N4FCM7UGXJWUIKG7DYINDDIO5L6KHBLTFXFTYNUYOFJTAOUA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC): protocol: rpc: dial tcp 104.36.82.16:4312: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:11.261871422 (13h33m10.851216409s) [noise=true , address=104.36.82.5:4866 , piece_id=XGJLHUY6KQX7JNJFYR2I7FX4FQGOAIXEFYOZISR246CR3UAKZWYA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: write tcp 188.166.225.170:34620->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:11.261899536 (13h33m10.851244523s) [seg_pos={0 0}, upload=14060]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:11.261973397 (13h33m10.851318374s) [upload=14060]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:11.261977566 (13h33m10.851322553s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.430348413 (13h37m26.0196934s) [upload=14060]: Done uploading segments... Aug 7 23:17:26.430376256 (13h37m26.019721243s) [upload=14060]: Upload finished. err: metaclient: context deadline exceeded Duration:* 15.688847283s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.491595337s FAIL (2025-08-07 23:17:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.491617127s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 3.195246097s FAIL (2025-08-07 23:17:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== segments... Aug 7 23:06:49.229716945 (13h26m48.819061932s) [upload=13950]: Upload finished. err: <nil> Aug 7 23:07:42.104963855 (13h27m41.694308842s) [upload=13968]: Starting upload Aug 7 23:07:42.105031205 (13h27m41.694376192s) [upload=13968]: Done starting upload Aug 7 23:07:42.105104063 (13h27m41.69444905s) [upload=13968]: Uploading segments... Aug 7 23:07:42.105139652 (13h27m41.694484639s) [upload=13968]: Got next segment. Inline: true Aug 7 23:07:42.105143259 (13h27m41.694488246s) [upload=13968]: Waiting for error group managing segments... Aug 7 23:07:42.105190129 (13h27m41.694535126s) [upload=13968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:42.105201120 (13h27m41.694546127s) [upload=13968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:42.105207281 (13h27m41.694552278s) [upload=13968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:42.417069033 (13h27m42.00641402s) [upload=13968]: Done uploading segments... Aug 7 23:07:42.417104130 (13h27m42.006449117s) [upload=13968]: Upload finished. err: <nil> Aug 7 23:08:34.230557646 (13h28m33.819902633s) [upload=13982]: Starting upload Aug 7 23:08:34.230604215 (13h28m33.819949193s) [upload=13982]: Done starting upload Aug 7 23:08:34.230655173 (13h28m33.82000016s) [upload=13982]: Uploading segments... Aug 7 23:08:34.230683928 (13h28m33.820028905s) [upload=13982]: Got next segment. Inline: true Aug 7 23:08:34.230687385 (13h28m33.820032362s) [upload=13982]: Waiting for error group managing segments... Aug 7 23:08:34.230719175 (13h28m33.820064162s) [upload=13982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:34.230722832 (13h28m33.820067819s) [upload=13982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:34.230725046 (13h28m33.820070033s) [upload=13982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:34.528494026 (13h28m34.117839013s) [upload=13982]: Done uploading segments... Aug 7 23:08:34.528512322 (13h28m34.117857309s) [upload=13982]: Upload finished. err: <nil> Aug 7 23:09:05.508570966 (13h29m5.097915953s) [upload=13989]: Starting upload Aug 7 23:09:05.508638426 (13h29m5.097983423s) [upload=13989]: Done starting upload Aug 7 23:09:05.508701747 (13h29m5.098046764s) [upload=13989]: Uploading segments... Aug 7 23:09:05.508743536 (13h29m5.098088533s) [upload=13989]: Got next segment. Inline: true Aug 7 23:09:05.508749488 (13h29m5.098094515s) [upload=13989]: Waiting for error group managing segments... Aug 7 23:09:05.508795345 (13h29m5.098140362s) [upload=13989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:05.508801848 (13h29m5.098146856s) [upload=13989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:05.508805736 (13h29m5.098150733s) [upload=13989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:05.806325549 (13h29m5.395670536s) [upload=13989]: Done uploading segments... Aug 7 23:09:05.806343342 (13h29m5.395688329s) [upload=13989]: Upload finished. err: <nil> Aug 7 23:09:38.090257337 (13h29m37.679602324s) [upload=14002]: Starting upload Aug 7 23:09:38.090300970 (13h29m37.679645957s) [upload=14002]: Done starting upload Aug 7 23:09:38.090372567 (13h29m37.679717554s) [upload=14002]: Uploading segments... Aug 7 23:09:38.090406682 (13h29m37.67975168s) [upload=14002]: Got next segment. Inline: true Aug 7 23:09:38.090410099 (13h29m37.679755076s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:09:38.090443282 (13h29m37.679788269s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:38.090447210 (13h29m37.679792198s) [upload=14002]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:38.090449415 (13h29m37.679794392s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:38.661400996 (13h29m38.250746043s) [upload=14002]: Done uploading segments... Aug 7 23:09:38.661423068 (13h29m38.250768135s) [upload=14002]: Upload finished. err: <nil> Aug 7 23:09:58.827549308 (13h29m58.416894295s) [upload=14009]: Starting upload Aug 7 23:09:58.827598442 (13h29m58.416943429s) [upload=14009]: Done starting upload Aug 7 23:09:58.827644020 (13h29m58.416989007s) [upload=14009]: Uploading segments... Aug 7 23:09:58.827674438 (13h29m58.417019425s) [upload=14009]: Got next segment. Inline: true Aug 7 23:09:58.827678135 (13h29m58.417023112s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:09:58.827711559 (13h29m58.417056546s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:58.827715526 (13h29m58.417060513s) [upload=14009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:58.827719804 (13h29m58.417064781s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:59.099262618 (13h29m58.688607605s) [upload=14009]: Done uploading segments... Aug 7 23:09:59.099279320 (13h29m58.688624297s) [upload=14009]: Upload finished. err: <nil> Aug 7 23:10:50.147486507 (13h30m49.736831494s) [upload=14023]: Starting upload Aug 7 23:10:50.147530131 (13h30m49.736875118s) [upload=14023]: Done starting upload Aug 7 23:10:50.147587851 (13h30m49.736932868s) [upload=14023]: Uploading segments... Aug 7 23:10:50.147650551 (13h30m49.736995568s) [upload=14023]: Got next segment. Inline: true Aug 7 23:10:50.147659338 (13h30m49.737004355s) [upload=14023]: Waiting for error group managing segments... Aug 7 23:10:50.147719804 (13h30m49.737064811s) [upload=14023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:50.147727708 (13h30m49.737072746s) [upload=14023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:50.147734752 (13h30m49.737079739s) [upload=14023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:50.421404977 (13h30m50.010749964s) [upload=14023]: Done uploading segments... Aug 7 23:10:50.421427519 (13h30m50.010772546s) [upload=14023]: Upload finished. err: <nil> Aug 7 23:11:30.587306004 (13h31m30.176650991s) [upload=14032]: Starting upload Aug 7 23:11:30.587348937 (13h31m30.176693924s) [upload=14032]: Done starting upload Aug 7 23:11:30.587398742 (13h31m30.17674373s) [upload=14032]: Uploading segments... Aug 7 23:11:30.587432447 (13h31m30.176777434s) [upload=14032]: Got next segment. Inline: true Aug 7 23:11:30.587437326 (13h31m30.176782313s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:11:30.587469467 (13h31m30.176814454s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:30.587472974 (13h31m30.176817951s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:30.587475168 (13h31m30.176820155s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:30.892529848 (13h31m30.481874835s) [upload=14032]: Done uploading segments... Aug 7 23:11:30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Duration: 3.195274711s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X.X....................................................XX] duration: 4m26.311955498s FAIL (2025-08-07 23:17:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad=14060]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:09.091387704 (13h33m8.680732691s) [seg_pos={0 0}, upload=14060]: Begin upload segment... Aug 7 23:13:09.091429043 (13h33m8.68077403s) [seg_pos={0 0}, upload=14060]: Done begin upload segment. Aug 7 23:13:09.091448139 (13h33m8.680793127s) [noise=true , address=104.36.82.35:4583 , piece_id=YCF4I7CDHUNPFY3GO5RJSSSELYU4GZFJOBCEIAUOZA5HGVV2AZIQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.091805804 (13h33m8.681150791s) [noise=true , address=104.36.82.5:4843 , piece_id=RFDJLWO26354TA2DOODQ5NWXQGBUCWSBVWNBZLNNHPBUMOQ5YG6A, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.092116358 (13h33m8.681461345s) [noise=true , address=104.36.82.59:4314 , piece_id=W6EMX4SZAY3KXXPV7FJNNZV6UXK2CMUPKB2S4E3PCT75R3P7CJBA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.092420690 (13h33m8.681765667s) [noise=true , address=104.36.82.16:4312 , piece_id=UD46N4FCM7UGXJWUIKG7DYINDDIO5L6KHBLTFXFTYNUYOFJTAOUA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.092787793 (13h33m8.68213278s) [noise=true , address=64.49.13.27:4964 , piece_id=3C44RWLK3JI7BW7II6WHJLY3KIK3FC3U4ZJHL47DJIKN3PVXDSFA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.093363383 (13h33m8.68270837s) [noise=true , address=45.91.116.8:4051 , piece_id=UOEY4RXDEGWOXOS3XHEZ2EZMAFJXNTQFO6ESPDXS3SP4EE4PTOTA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.093583274 (13h33m8.682928261s) [noise=true , address=104.36.82.5:4866 , piece_id=XGJLHUY6KQX7JNJFYR2I7FX4FQGOAIXEFYOZISR246CR3UAKZWYA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.093789509 (13h33m8.683134496s) [noise=true , address=104.36.82.52:4095 , piece_id=72VSSUO6C6ZJ6RJ627ZL6WNWBPKUGTPBWI6GO44ZJFDQYEFRLYLQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.094036702 (13h33m8.683381689s) [noise=true , address=104.36.82.45:4932 , piece_id=RDVHKOUB3MFQXCWQYIP2GTYGLE4A5JORDQO54WWPFERQG7W22ZLQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.094306238 (13h33m8.683651225s) [noise=true , address=79.127.201.215:4347 , piece_id=ICE4FM3Z2PFPLG3VQEXADCCVALDUFBX7RBAAABIBGSSA3BDJBABQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.094581394 (13h33m8.683926381s) [noise=true , address=64.49.13.169:4713 , piece_id=OC4GKJQXE7JXL6J4YNCJERNCDGFA25FDPZ3EOMUJ6JWCXJUTZZXA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.094890105 (13h33m8.684235082s) [noise=true , address=64.49.13.137:4493 , piece_id=WC4R5BUB46HUXV3XKARPMEWWWLXJV5LJ6QNLXIOEWCZBXJYYA6BQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.095243872 (13h33m8.684588849s) [noise=true , address=104.36.82.39:4749 , piece_id=4SYI2SFT6YSMDTOTXJTINJ7LU2LD4ZHAOME4ZZXVKU5CGBNPG6BQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.095576859 (13h33m8.684921846s) [noise=true , address=104.36.82.19:4408 , piece_id=HQQTSW7FJVUAZPVJ5DXFK35RN2LVQAIFOVG3NHVLFL3WRL7AFPHQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.095852887 (13h33m8.685197874s) [noise=true , address=45.91.116.11:4151 , piece_id=KSXLHRUPUUUK4UVIF76ASVMQEUTSIXMNTB5WMZ52NRAE2ZF6M5BQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.096045065 (13h33m8.685390052s) [noise=true , address=104.36.82.43:4863 , piece_id=7ETQFK4AWX2OQOPCQ32OH6VJUAS46GTYXYMNM2Y5JAGRD7NOSMWQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.096498421 (13h33m8.685843409s) [noise=true , address=104.36.82.44:4901 , piece_id=5GNUWDIXFSJLG4QJW7IS5MD2W35YMQMGRYZ5G6JWP2XL43AG76EA, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:09.096818584 (13h33m8.686163571s) [noise=true , address=104.36.82.36:4646 , piece_id=X374LZM3CKNA3PNJ5F4W6KMUQELJWKWAPWXHPNBBPQC4DYJVLWWQ, seg_pos={0 0}, upload=14060]: Uploading piece... Aug 7 23:13:10.198767377 (13h33m9.788112364s) [upload=14060]: Next returned nil segment Aug 7 23:13:10.198784199 (13h33m9.788129186s) [upload=14060]: Waiting for error group managing segments... Aug 7 23:13:10.482190972 (13h33m10.071535949s) [noise=true , address=64.49.13.137:4493 , piece_id=WC4R5BUB46HUXV3XKARPMEWWWLXJV5LJ6QNLXIOEWCZBXJYYA6BQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:10.513415901 (13h33m10.102760888s) [noise=true , address=64.49.13.27:4964 , piece_id=3C44RWLK3JI7BW7II6WHJLY3KIK3FC3U4ZJHL47DJIKN3PVXDSFA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:10.515976149 (13h33m10.105321136s) [noise=true , address=64.49.13.169:4713 , piece_id=OC4GKJQXE7JXL6J4YNCJERNCDGFA25FDPZ3EOMUJ6JWCXJUTZZXA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.003898079 (13h33m10.593243066s) [noise=true , address=104.36.82.39:4749 , piece_id=4SYI2SFT6YSMDTOTXJTINJ7LU2LD4ZHAOME4ZZXVKU5CGBNPG6BQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.038920043 (13h33m10.62826503s) [noise=true , address=79.127.201.215:4347 , piece_id=ICE4FM3Z2PFPLG3VQEXADCCVALDUFBX7RBAAABIBGSSA3BDJBABQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.061211160 (13h33m10.650556157s) [noise=true , address=104.36.82.43:4863 , piece_id=7ETQFK4AWX2OQOPCQ32OH6VJUAS46GTYXYMNM2Y5JAGRD7NOSMWQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.116268806 (13h33m10.705613793s) [noise=true , address=104.36.82.19:4408 , piece_id=HQQTSW7FJVUAZPVJ5DXFK35RN2LVQAIFOVG3NHVLFL3WRL7AFPHQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.119226344 (13h33m10.708571381s) [noise=true , address=104.36.82.45:4932 , piece_id=RDVHKOUB3MFQXCWQYIP2GTYGLE4A5JORDQO54WWPFERQG7W22ZLQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.163437749 (13h33m10.752782736s) [noise=true , address=104.36.82.44:4901 , piece_id=5GNUWDIXFSJLG4QJW7IS5MD2W35YMQMGRYZ5G6JWP2XL43AG76EA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.166785074 (13h33m10.756130051s) [noise=true , address=104.36.82.36:4646 , piece_id=X374LZM3CKNA3PNJ5F4W6KMUQELJWKWAPWXHPNBBPQC4DYJVLWWQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.190978580 (13h33m10.780323557s) [noise=true , address=104.36.82.35:4583 , piece_id=YCF4I7CDHUNPFY3GO5RJSSSELYU4GZFJOBCEIAUOZA5HGVV2AZIQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.229816815 (13h33m10.819161802s) [noise=true , address=45.91.116.8:4051 , piece_id=UOEY4RXDEGWOXOS3XHEZ2EZMAFJXNTQFO6ESPDXS3SP4EE4PTOTA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.257926594 (13h33m10.847271581s) [noise=true , address=45.91.116.11:4151 , piece_id=KSXLHRUPUUUK4UVIF76ASVMQEUTSIXMNTB5WMZ52NRAE2ZF6M5BQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.261359843 (13h33m10.85070483s) [noise=true , address=104.36.82.52:4095 , piece_id=72VSSUO6C6ZJ6RJ627ZL6WNWBPKUGTPBWI6GO44ZJFDQYEFRLYLQ, seg_pos={0 0}, upload=14060]: Done uploading piece. err: <nil> Aug 7 23:13:11.261376014 (13h33m10.850721002s) [seg_pos={0 0}, upload=14060]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:11.261727868 (13h33m10.851072855s) [noise=true , address=104.36.82.5:4843 , piece_id=RFDJLWO26354TA2DOODQ5NWXQGBUCWSBVWNBZLNNHPBUMOQ5YG6A, seg_pos={0 0}, upload=14060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HvL2JDP4GCsu3tziVEKRMh14nEzyRMRuWNSmVvs27LzHTbNpS): protocol: write tcp 188.166.225.170:37258->104.36.82.5:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:11.261771851 (13h33m10.851116838s) [noise=true , address=104.36.82.59:4314 , piece_id=W6EMX4SZAY3KXXPV7FJNNZV6UXK2CMUPKB2S4E3PCT75R3P7CJBA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EsmL66Gn6K9cUxHqqGtuUZxUGyWCNL4oqyHzS1XUbWVnMMJbd): protocol: write tcp 188.166.225.170:54566->104.36.82.59:4314: use of closed network connection; write tcp 188.166.225.170:54566->104.36.82.59:4314: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54566->104.36.82.59:4314: use of closed network connection Aug 7 23:13:11.261778304 (13h33m10.851123291s) [noise=true , address=104.36.82.16:4312 , piece_id=UD46N4FCM7UGXJWUIKG7DYINDDIO5L6KHBLTFXFTYNUYOFJTAOUA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC): protocol: rpc: dial tcp 104.36.82.16:4312: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:11.261871422 (13h33m10.851216409s) [noise=true , address=104.36.82.5:4866 , piece_id=XGJLHUY6KQX7JNJFYR2I7FX4FQGOAIXEFYOZISR246CR3UAKZWYA, seg_pos={0 0}, upload=14060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: write tcp 188.166.225.170:34620->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:11.261899536 (13h33m10.851244523s) [seg_pos={0 0}, upload=14060]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:11.261973397 (13h33m10.851318374s) [upload=14060]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:11.261977566 (13h33m10.851322553s) [upload=14060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.430348413 (13h37m26.0196934s) [upload=14060]: Done uploading segments... Aug 7 23:17:26.430376256 (13h37m26.019721243s) [upload=14060]: Upload finished. err: metaclient: context deadline exceeded Duration:* 4m26.311968392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXXXX..] duration: 1.921974392s FAIL (2025-08-07 23:17:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.921998378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXX.X] duration: 30.001266271s FAIL (2025-08-07 23:17:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001285426s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................X..X.XXX] duration: 30.001671293s FAIL (2025-08-07 23:17:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Done uploading segments... Aug 7 23:14:41.331082898 (13h34m40.920427885s) [upload=14078]: Upload finished. err: context deadline exceeded Aug 7 23:14:54.150263358 (13h34m53.739608355s) [upload=14088]: Starting upload Aug 7 23:14:54.150316199 (13h34m53.739661186s) [upload=14088]: Done starting upload Aug 7 23:14:54.150403316 (13h34m53.739748353s) [upload=14088]: Uploading segments... Aug 7 23:14:54.150514368 (13h34m53.739859345s) [upload=14088]: Got next segment. Inline: false Aug 7 23:14:54.150558122 (13h34m53.739903119s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:54.150565516 (13h34m53.739910543s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:58.174360057 (13h34m57.763705044s) [seg_pos={0 0}, upload=14088]: Begin upload segment... Aug 7 23:14:58.174405564 (13h34m57.763750551s) [seg_pos={0 0}, upload=14088]: Done begin upload segment. Aug 7 23:14:58.174494114 (13h34m57.763839101s) [noise=true , address=104.36.82.4:4801 , piece_id=PWV7TZNRT5SV5MDH4F3LVVZYPPY5E53F2QB6D3APCET36JQ7G2WQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174594706 (13h34m57.763939693s) [noise=true , address=77.90.31.17:4771 , piece_id=OBAQRZSSIMSEBMGQ3OVE7BVT5L6BRAST4SU4YEL2HQR3TB4C7RWA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174917945 (13h34m57.764262922s) [noise=true , address=104.36.82.11:4113 , piece_id=ZFACTZDEWA2A2ILT47CFS4DXXVCCUXOJ3FAKN3GCOQW4PBQDGP7A, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174963081 (13h34m57.764308068s) [noise=true , address=77.90.31.13:4627 , piece_id=ZPVXMNQC6C7CDYVTPQO4ZWHRT6H4G643QHZXLSMTY3ROAQDA2PAQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175227337 (13h34m57.764572314s) [noise=true , address=45.91.116.13:4154 , piece_id=LK6FRJB4SOGTCSDGBMO7M2UO532GHS3YSBB3UBQJ5MQQSOKRNBKA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175234811 (13h34m57.764579798s) [noise=true , address=104.36.82.15:4257 , piece_id=QO4HU2TMEQJVVT4IJWQBXFIHWWYC3RNO67BHEH2D3M3COEO5WGHA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175498105 (13h34m57.764843092s) [noise=true , address=77.90.31.18:4806 , piece_id=3ZQ5IIU34KBAMPDBA676VXYJNVCJLDJ2LJEMGQUVUPU4SL55C5TQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175523783 (13h34m57.76486877s) [noise=true , address=45.91.116.11:4117 , piece_id=DPVFTWIVMADRY4KGDQHM7VEMCMZCME56WS3E7D2SJM36XGZ5ERBA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175757932 (13h34m57.765102919s) [noise=true , address=45.91.116.14:4190 , piece_id=DE6K7C7L2SCIWO3OH2DPBUVMECGTDSUUGKZQLQ2PA22J4B4RPR5Q, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175784072 (13h34m57.765129049s) [noise=true , address=104.36.82.9:4006 , piece_id=G5E2Y462PS3JPXHMF2OABO6IJL27KLK34USATI7AY5GV2SSPYKHQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176054940 (13h34m57.765399927s) [noise=true , address=77.90.31.11:4553 , piece_id=2TEUIH35CQDRQY6KMY4KRXHOIV2RRMB5WRTOHW56M7BG7BY4WFZQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176240194 (13h34m57.765585181s) [noise=true , address=45.91.116.5:4297 , piece_id=V7BR6TM3LQM3ERWNGT3O7KYJV4XNVWYGKASOWDG5NPJ6HMSGLAAA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176396734 (13h34m57.765741721s) [noise=true , address=77.90.31.16:4735 , piece_id=ATYURAKL6O2SFDK4ARD5ERDIZGVTJ34F5JBBULRNYETGFRSOUCMQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176542252 (13h34m57.765887239s) [noise=true , address=45.91.116.10:4082 , piece_id=SCGMY34O4OZF4FC63ZSE3KPDK4A2AF55ASQOPQETC73GCCJASCZA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176624790 (13h34m57.765969777s) [noise=true , address=45.91.116.9:4262 , piece_id=7CSV4KG24RRZOTWHO4HBBFCOHBR7WHFSWYI2JQ7D7GQVRO5QPTWQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176827989 (13h34m57.766172966s) [noise=true , address=77.90.31.21:4878 , piece_id=RDAB6QNSRNPWMNFASKEKGHLUWSTS5AQGV5MZU5EW7F7WTX4RBIRA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:59.945982215 (13h34m59.535327192s) [upload=14088]: Next returned nil segment Aug 7 23:14:59.946000040 (13h34m59.535345027s) [upload=14088]: Waiting for error group managing segments... Aug 7 23:15:00.235597016 (13h34m59.824942003s) [noise=true , address=104.36.82.4:4801 , piece_id=PWV7TZNRT5SV5MDH4F3LVVZYPPY5E53F2QB6D3APCET36JQ7G2WQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.336486456 (13h34m59.925831443s) [noise=true , address=104.36.82.9:4006 , piece_id=G5E2Y462PS3JPXHMF2OABO6IJL27KLK34USATI7AY5GV2SSPYKHQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.461953790 (13h35m0.051298777s) [noise=true , address=104.36.82.15:4257 , piece_id=QO4HU2TMEQJVVT4IJWQBXFIHWWYC3RNO67BHEH2D3M3COEO5WGHA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.536493809 (13h35m0.125838786s) [noise=true , address=77.90.31.17:4771 , piece_id=OBAQRZSSIMSEBMGQ3OVE7BVT5L6BRAST4SU4YEL2HQR3TB4C7RWA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.547607852 (13h35m0.13695284s) [noise=true , address=77.90.31.16:4735 , piece_id=ATYURAKL6O2SFDK4ARD5ERDIZGVTJ34F5JBBULRNYETGFRSOUCMQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.557092168 (13h35m0.146437155s) [noise=true , address=45.91.116.9:4262 , piece_id=7CSV4KG24RRZOTWHO4HBBFCOHBR7WHFSWYI2JQ7D7GQVRO5QPTWQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.588663019 (13h35m0.178007996s) [noise=true , address=77.90.31.11:4553 , piece_id=2TEUIH35CQDRQY6KMY4KRXHOIV2RRMB5WRTOHW56M7BG7BY4WFZQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.669587042 (13h35m0.258932019s) [noise=true , address=104.36.82.11:4113 , piece_id=ZFACTZDEWA2A2ILT47CFS4DXXVCCUXOJ3FAKN3GCOQW4PBQDGP7A, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.772791433 (13h35m0.36213642s) [noise=true , address=45.91.116.11:4117 , piece_id=DPVFTWIVMADRY4KGDQHM7VEMCMZCME56WS3E7D2SJM36XGZ5ERBA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.788386820 (13h35m0.377731797s) [noise=true , address=77.90.31.13:4627 , piece_id=ZPVXMNQC6C7CDYVTPQO4ZWHRT6H4G643QHZXLSMTY3ROAQDA2PAQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.856809245 (13h35m0.446154232s) [noise=true , address=45.91.116.10:4082 , piece_id=SCGMY34O4OZF4FC63ZSE3KPDK4A2AF55ASQOPQETC73GCCJASCZA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.865234375 (13h35m0.454579362s) [noise=true , address=45.91.116.14:4190 , piece_id=DE6K7C7L2SCIWO3OH2DPBUVMECGTDSUUGKZQLQ2PA22J4B4RPR5Q, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.886019812 (13h35m0.475364799s) [noise=true , address=77.90.31.21:4878 , piece_id=RDAB6QNSRNPWMNFASKEKGHLUWSTS5AQGV5MZU5EW7F7WTX4RBIRA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.886047264 (13h35m0.475392241s) [seg_pos={0 0}, upload=14088]: Segment reached optimal threshold of 13 pieces. Aug 7 23:15:00.886165169 (13h35m0.475510156s) [noise=true , address=45.91.116.13:4154 , piece_id=LK6FRJB4SOGTCSDGBMO7M2UO532GHS3YSBB3UBQJ5MQQSOKRNBKA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886224553 (13h35m0.47556954s) [noise=true , address=45.91.116.5:4297 , piece_id=V7BR6TM3LQM3ERWNGT3O7KYJV4XNVWYGKASOWDG5NPJ6HMSGLAAA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886291521 (13h35m0.475636498s) [noise=true , address=77.90.31.18:4806 , piece_id=3ZQ5IIU34KBAMPDBA676VXYJNVCJLDJ2LJEMGQUVUPU4SL55C5TQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886340935 (13h35m0.475685922s) [seg_pos={0 0}, upload=14088]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:15:00.886432311 (13h35m0.475777298s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:15:00.886436909 (13h35m0.475781896s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:49.547592064 (13h36m49.136937041s) [upload=14088]: Done uploading segments... Aug 7 23:16:49.547614097 (13h36m49.136959084s) [upload=14088]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:50.417681936 (13h36m50.007026923s) [upload=14116]: Starting upload Aug 7 23:16:50.417744316 (13h36m50.007089294s) [upload=14116]: Done starting upload Aug 7 23:16:50.417844638 (13h36m50.007189625s) [upload=14116]: Uploading segments... Aug 7 23:16:50.417958475 (13h36m50.007303462s) [upload=14116]: Got next segment. Inline: false Aug 7 23:16:50.417989144 (13h36m50.007334131s) [upload=14116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:50.417993503 (13h36m50.00733849s) [upload=14116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:17:19.777322561 (13h37m19.366667548s) [upload=14116]: Next segment err: context deadline exceeded Aug 7 23:17:19.777345936 (13h37m19.366690923s) [upload=14116]: Done uploading segments... Aug 7 23:17:19.777363158 (13h37m19.366708166s) [upload=14116]: Upload finished. err: context deadline exceeded Duration: 30.001722342s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 646.346647ms target: https://us1.storj.io/login FAIL (2025-08-07 23:17:13 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 646.369682ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.000662196s FAIL (2025-08-07 23:17:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ed. err: <nil> Aug 7 23:09:35.163599610 (13h29m34.752944597s) [upload=14001]: Starting upload Aug 7 23:09:35.163659795 (13h29m34.753004782s) [upload=14001]: Done starting upload Aug 7 23:09:35.163720571 (13h29m34.753065558s) [upload=14001]: Uploading segments... Aug 7 23:09:35.163772821 (13h29m34.753117808s) [upload=14001]: Got next segment. Inline: true Aug 7 23:09:35.163777159 (13h29m34.753122146s) [upload=14001]: Waiting for error group managing segments... Aug 7 23:09:35.163816885 (13h29m34.753161872s) [upload=14001]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:35.163830291 (13h29m34.753175278s) [upload=14001]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:35.163832876 (13h29m34.753177863s) [upload=14001]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:35.429570925 (13h29m35.018915942s) [upload=14001]: Done uploading segments... Aug 7 23:09:35.429591264 (13h29m35.018936251s) [upload=14001]: Upload finished. err: <nil> Aug 7 23:09:54.175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000682515s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.350603092s FAIL (2025-08-07 23:17:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.35062336s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration: 1.888566576s FAIL (2025-08-07 23:17:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.88858421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.000768066s FAIL (2025-08-07 23:17:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000778956s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X.....................................................XXXXX] duration: 1m0.000807423s FAIL (2025-08-07 23:16:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00081665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................X..X.XX] duration: 1m56.044079285s FAIL (2025-08-07 23:16:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 96898s) [upload=14073]: Done uploading segments... Aug 7 23:13:55.438270216 (13h33m55.027615203s) [upload=14073]: Upload finished. err: <nil> Aug 7 23:14:11.982803233 (13h34m11.57214823s) [upload=14078]: Starting upload Aug 7 23:14:11.982897995 (13h34m11.572242982s) [upload=14078]: Done starting upload Aug 7 23:14:11.982982667 (13h34m11.572327654s) [upload=14078]: Uploading segments... Aug 7 23:14:11.983268052 (13h34m11.57261304s) [upload=14078]: Got next segment. Inline: false Aug 7 23:14:11.983335101 (13h34m11.572680098s) [upload=14078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:11.983341213 (13h34m11.5726862s) [upload=14078]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:41.331003236 (13h34m40.920348223s) [upload=14078]: Next segment err: context deadline exceeded Aug 7 23:14:41.331068089 (13h34m40.920413076s) [upload=14078]: Done uploading segments... Aug 7 23:14:41.331082898 (13h34m40.920427885s) [upload=14078]: Upload finished. err: context deadline exceeded Aug 7 23:14:54.150263358 (13h34m53.739608355s) [upload=14088]: Starting upload Aug 7 23:14:54.150316199 (13h34m53.739661186s) [upload=14088]: Done starting upload Aug 7 23:14:54.150403316 (13h34m53.739748353s) [upload=14088]: Uploading segments... Aug 7 23:14:54.150514368 (13h34m53.739859345s) [upload=14088]: Got next segment. Inline: false Aug 7 23:14:54.150558122 (13h34m53.739903119s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:54.150565516 (13h34m53.739910543s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:14:58.174360057 (13h34m57.763705044s) [seg_pos={0 0}, upload=14088]: Begin upload segment... Aug 7 23:14:58.174405564 (13h34m57.763750551s) [seg_pos={0 0}, upload=14088]: Done begin upload segment. Aug 7 23:14:58.174494114 (13h34m57.763839101s) [noise=true , address=104.36.82.4:4801 , piece_id=PWV7TZNRT5SV5MDH4F3LVVZYPPY5E53F2QB6D3APCET36JQ7G2WQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174594706 (13h34m57.763939693s) [noise=true , address=77.90.31.17:4771 , piece_id=OBAQRZSSIMSEBMGQ3OVE7BVT5L6BRAST4SU4YEL2HQR3TB4C7RWA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174917945 (13h34m57.764262922s) [noise=true , address=104.36.82.11:4113 , piece_id=ZFACTZDEWA2A2ILT47CFS4DXXVCCUXOJ3FAKN3GCOQW4PBQDGP7A, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.174963081 (13h34m57.764308068s) [noise=true , address=77.90.31.13:4627 , piece_id=ZPVXMNQC6C7CDYVTPQO4ZWHRT6H4G643QHZXLSMTY3ROAQDA2PAQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175227337 (13h34m57.764572314s) [noise=true , address=45.91.116.13:4154 , piece_id=LK6FRJB4SOGTCSDGBMO7M2UO532GHS3YSBB3UBQJ5MQQSOKRNBKA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175234811 (13h34m57.764579798s) [noise=true , address=104.36.82.15:4257 , piece_id=QO4HU2TMEQJVVT4IJWQBXFIHWWYC3RNO67BHEH2D3M3COEO5WGHA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175498105 (13h34m57.764843092s) [noise=true , address=77.90.31.18:4806 , piece_id=3ZQ5IIU34KBAMPDBA676VXYJNVCJLDJ2LJEMGQUVUPU4SL55C5TQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175523783 (13h34m57.76486877s) [noise=true , address=45.91.116.11:4117 , piece_id=DPVFTWIVMADRY4KGDQHM7VEMCMZCME56WS3E7D2SJM36XGZ5ERBA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175757932 (13h34m57.765102919s) [noise=true , address=45.91.116.14:4190 , piece_id=DE6K7C7L2SCIWO3OH2DPBUVMECGTDSUUGKZQLQ2PA22J4B4RPR5Q, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.175784072 (13h34m57.765129049s) [noise=true , address=104.36.82.9:4006 , piece_id=G5E2Y462PS3JPXHMF2OABO6IJL27KLK34USATI7AY5GV2SSPYKHQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176054940 (13h34m57.765399927s) [noise=true , address=77.90.31.11:4553 , piece_id=2TEUIH35CQDRQY6KMY4KRXHOIV2RRMB5WRTOHW56M7BG7BY4WFZQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176240194 (13h34m57.765585181s) [noise=true , address=45.91.116.5:4297 , piece_id=V7BR6TM3LQM3ERWNGT3O7KYJV4XNVWYGKASOWDG5NPJ6HMSGLAAA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176396734 (13h34m57.765741721s) [noise=true , address=77.90.31.16:4735 , piece_id=ATYURAKL6O2SFDK4ARD5ERDIZGVTJ34F5JBBULRNYETGFRSOUCMQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176542252 (13h34m57.765887239s) [noise=true , address=45.91.116.10:4082 , piece_id=SCGMY34O4OZF4FC63ZSE3KPDK4A2AF55ASQOPQETC73GCCJASCZA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176624790 (13h34m57.765969777s) [noise=true , address=45.91.116.9:4262 , piece_id=7CSV4KG24RRZOTWHO4HBBFCOHBR7WHFSWYI2JQ7D7GQVRO5QPTWQ, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:58.176827989 (13h34m57.766172966s) [noise=true , address=77.90.31.21:4878 , piece_id=RDAB6QNSRNPWMNFASKEKGHLUWSTS5AQGV5MZU5EW7F7WTX4RBIRA, seg_pos={0 0}, upload=14088]: Uploading piece... Aug 7 23:14:59.945982215 (13h34m59.535327192s) [upload=14088]: Next returned nil segment Aug 7 23:14:59.946000040 (13h34m59.535345027s) [upload=14088]: Waiting for error group managing segments... Aug 7 23:15:00.235597016 (13h34m59.824942003s) [noise=true , address=104.36.82.4:4801 , piece_id=PWV7TZNRT5SV5MDH4F3LVVZYPPY5E53F2QB6D3APCET36JQ7G2WQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.336486456 (13h34m59.925831443s) [noise=true , address=104.36.82.9:4006 , piece_id=G5E2Y462PS3JPXHMF2OABO6IJL27KLK34USATI7AY5GV2SSPYKHQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.461953790 (13h35m0.051298777s) [noise=true , address=104.36.82.15:4257 , piece_id=QO4HU2TMEQJVVT4IJWQBXFIHWWYC3RNO67BHEH2D3M3COEO5WGHA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.536493809 (13h35m0.125838786s) [noise=true , address=77.90.31.17:4771 , piece_id=OBAQRZSSIMSEBMGQ3OVE7BVT5L6BRAST4SU4YEL2HQR3TB4C7RWA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.547607852 (13h35m0.13695284s) [noise=true , address=77.90.31.16:4735 , piece_id=ATYURAKL6O2SFDK4ARD5ERDIZGVTJ34F5JBBULRNYETGFRSOUCMQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.557092168 (13h35m0.146437155s) [noise=true , address=45.91.116.9:4262 , piece_id=7CSV4KG24RRZOTWHO4HBBFCOHBR7WHFSWYI2JQ7D7GQVRO5QPTWQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.588663019 (13h35m0.178007996s) [noise=true , address=77.90.31.11:4553 , piece_id=2TEUIH35CQDRQY6KMY4KRXHOIV2RRMB5WRTOHW56M7BG7BY4WFZQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.669587042 (13h35m0.258932019s) [noise=true , address=104.36.82.11:4113 , piece_id=ZFACTZDEWA2A2ILT47CFS4DXXVCCUXOJ3FAKN3GCOQW4PBQDGP7A, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.772791433 (13h35m0.36213642s) [noise=true , address=45.91.116.11:4117 , piece_id=DPVFTWIVMADRY4KGDQHM7VEMCMZCME56WS3E7D2SJM36XGZ5ERBA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.788386820 (13h35m0.377731797s) [noise=true , address=77.90.31.13:4627 , piece_id=ZPVXMNQC6C7CDYVTPQO4ZWHRT6H4G643QHZXLSMTY3ROAQDA2PAQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.856809245 (13h35m0.446154232s) [noise=true , address=45.91.116.10:4082 , piece_id=SCGMY34O4OZF4FC63ZSE3KPDK4A2AF55ASQOPQETC73GCCJASCZA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.865234375 (13h35m0.454579362s) [noise=true , address=45.91.116.14:4190 , piece_id=DE6K7C7L2SCIWO3OH2DPBUVMECGTDSUUGKZQLQ2PA22J4B4RPR5Q, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.886019812 (13h35m0.475364799s) [noise=true , address=77.90.31.21:4878 , piece_id=RDAB6QNSRNPWMNFASKEKGHLUWSTS5AQGV5MZU5EW7F7WTX4RBIRA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: <nil> Aug 7 23:15:00.886047264 (13h35m0.475392241s) [seg_pos={0 0}, upload=14088]: Segment reached optimal threshold of 13 pieces. Aug 7 23:15:00.886165169 (13h35m0.475510156s) [noise=true , address=45.91.116.13:4154 , piece_id=LK6FRJB4SOGTCSDGBMO7M2UO532GHS3YSBB3UBQJ5MQQSOKRNBKA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886224553 (13h35m0.47556954s) [noise=true , address=45.91.116.5:4297 , piece_id=V7BR6TM3LQM3ERWNGT3O7KYJV4XNVWYGKASOWDG5NPJ6HMSGLAAA, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886291521 (13h35m0.475636498s) [noise=true , address=77.90.31.18:4806 , piece_id=3ZQ5IIU34KBAMPDBA676VXYJNVCJLDJ2LJEMGQUVUPU4SL55C5TQ, seg_pos={0 0}, upload=14088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Aug 7 23:15:00.886340935 (13h35m0.475685922s) [seg_pos={0 0}, upload=14088]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:15:00.886432311 (13h35m0.475777298s) [upload=14088]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:15:00.886436909 (13h35m0.475781896s) [upload=14088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:49.547592064 (13h36m49.136937041s) [upload=14088]: Done uploading segments... Aug 7 23:16:49.547614097 (13h36m49.136959084s) [upload=14088]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m56.044101576s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 648.531218ms target: https://us1.storj.io/login FAIL (2025-08-07 23:16:45 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 648.553019ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.240255715s FAIL (2025-08-07 23:16:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.240277217s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 1.908553947s FAIL (2025-08-07 23:16:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.908584706s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X...................X....................X....] duration: 6.630515468s FAIL (2025-08-07 23:16:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e85db788dbeec38ad8084daa355e1bf59b8b718af998fc40c7451ca1ba6627b2 != before 1cad814b0341363008cdb7d22dd66bb6d87f0105cd39850888ced4405a0b4b43 ======LOGS====== upload Aug 7 23:13:43.967401320 (13h33m43.556746307s) [upload=14069]: Uploading segments... Aug 7 23:13:43.967498206 (13h33m43.556843193s) [upload=14069]: Got next segment. Inline: false Aug 7 23:13:43.967560455 (13h33m43.556905442s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:43.967568550 (13h33m43.556913577s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:44.414536988 (13h33m44.003881975s) [seg_pos={0 0}, upload=14069]: Begin upload segment... Aug 7 23:13:44.414587104 (13h33m44.003932091s) [seg_pos={0 0}, upload=14069]: Done begin upload segment. Aug 7 23:13:44.414606933 (13h33m44.00395192s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414666236 (13h33m44.004011223s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414966160 (13h33m44.004311147s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415118381 (13h33m44.004463369s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415228682 (13h33m44.00457367s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415496494 (13h33m44.004841471s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415745370 (13h33m44.005090357s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416026788 (13h33m44.005371766s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416256569 (13h33m44.005601556s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416456080 (13h33m44.005801067s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416552746 (13h33m44.005897753s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416803596 (13h33m44.006148583s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417042221 (13h33m44.006387208s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417277101 (13h33m44.006622088s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417316085 (13h33m44.006661072s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417537459 (13h33m44.006882436s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417738975 (13h33m44.007083952s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417952763 (13h33m44.00729774s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:45.514309387 (13h33m45.103654374s) [upload=14069]: Next returned nil segment Aug 7 23:13:45.514327020 (13h33m45.103672007s) [upload=14069]: Waiting for error group managing segments... Aug 7 23:13:45.795258156 (13h33m45.384603143s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.802263469 (13h33m45.391608456s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.887877424 (13h33m45.477222401s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.350209185 (13h33m45.939554172s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.389672887 (13h33m45.979017874s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.399409194 (13h33m45.988754171s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.424971119 (13h33m46.014316106s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.450918913 (13h33m46.0402639s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.492609174 (13h33m46.081954161s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.516716376 (13h33m46.106061353s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.592315332 (13h33m46.181660319s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.746385819 (13h33m46.335730806s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.786539771 (13h33m46.375884758s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838001699 (13h33m46.427346676s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838017648 (13h33m46.427362636s) [seg_pos={0 0}, upload=14069]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:46.838217681 (13h33m46.427562658s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838422624 (13h33m46.427767601s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838575365 (13h33m46.427920352s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: rpc: dial tcp 104.36.82.16:4296: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838627195 (13h33m46.427972172s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:48060->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838635551 (13h33m46.427980538s) [seg_pos={0 0}, upload=14069]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:46.838702128 (13h33m46.428047115s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:46.838705895 (13h33m46.428050882s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.127771955 (13h33m46.717116932s) [upload=14069]: Done uploading segments... Aug 7 23:13:47.127791082 (13h33m46.717136069s) [upload=14069]: Upload finished. err: <nil> Duration: 6.630543462s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.845639707s FAIL (2025-08-07 23:16:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== segments... Aug 7 23:06:49.229716945 (13h26m48.819061932s) [upload=13950]: Upload finished. err: <nil> Aug 7 23:07:42.104963855 (13h27m41.694308842s) [upload=13968]: Starting upload Aug 7 23:07:42.105031205 (13h27m41.694376192s) [upload=13968]: Done starting upload Aug 7 23:07:42.105104063 (13h27m41.69444905s) [upload=13968]: Uploading segments... Aug 7 23:07:42.105139652 (13h27m41.694484639s) [upload=13968]: Got next segment. Inline: true Aug 7 23:07:42.105143259 (13h27m41.694488246s) [upload=13968]: Waiting for error group managing segments... Aug 7 23:07:42.105190129 (13h27m41.694535126s) [upload=13968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:42.105201120 (13h27m41.694546127s) [upload=13968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:42.105207281 (13h27m41.694552278s) [upload=13968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:42.417069033 (13h27m42.00641402s) [upload=13968]: Done uploading segments... Aug 7 23:07:42.417104130 (13h27m42.006449117s) [upload=13968]: Upload finished. err: <nil> Aug 7 23:08:34.230557646 (13h28m33.819902633s) [upload=13982]: Starting upload Aug 7 23:08:34.230604215 (13h28m33.819949193s) [upload=13982]: Done starting upload Aug 7 23:08:34.230655173 (13h28m33.82000016s) [upload=13982]: Uploading segments... Aug 7 23:08:34.230683928 (13h28m33.820028905s) [upload=13982]: Got next segment. Inline: true Aug 7 23:08:34.230687385 (13h28m33.820032362s) [upload=13982]: Waiting for error group managing segments... Aug 7 23:08:34.230719175 (13h28m33.820064162s) [upload=13982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:34.230722832 (13h28m33.820067819s) [upload=13982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:34.230725046 (13h28m33.820070033s) [upload=13982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:34.528494026 (13h28m34.117839013s) [upload=13982]: Done uploading segments... Aug 7 23:08:34.528512322 (13h28m34.117857309s) [upload=13982]: Upload finished. err: <nil> Aug 7 23:09:05.508570966 (13h29m5.097915953s) [upload=13989]: Starting upload Aug 7 23:09:05.508638426 (13h29m5.097983423s) [upload=13989]: Done starting upload Aug 7 23:09:05.508701747 (13h29m5.098046764s) [upload=13989]: Uploading segments... Aug 7 23:09:05.508743536 (13h29m5.098088533s) [upload=13989]: Got next segment. Inline: true Aug 7 23:09:05.508749488 (13h29m5.098094515s) [upload=13989]: Waiting for error group managing segments... Aug 7 23:09:05.508795345 (13h29m5.098140362s) [upload=13989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:05.508801848 (13h29m5.098146856s) [upload=13989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:05.508805736 (13h29m5.098150733s) [upload=13989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:05.806325549 (13h29m5.395670536s) [upload=13989]: Done uploading segments... Aug 7 23:09:05.806343342 (13h29m5.395688329s) [upload=13989]: Upload finished. err: <nil> Aug 7 23:09:38.090257337 (13h29m37.679602324s) [upload=14002]: Starting upload Aug 7 23:09:38.090300970 (13h29m37.679645957s) [upload=14002]: Done starting upload Aug 7 23:09:38.090372567 (13h29m37.679717554s) [upload=14002]: Uploading segments... Aug 7 23:09:38.090406682 (13h29m37.67975168s) [upload=14002]: Got next segment. Inline: true Aug 7 23:09:38.090410099 (13h29m37.679755076s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:09:38.090443282 (13h29m37.679788269s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:38.090447210 (13h29m37.679792198s) [upload=14002]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:38.090449415 (13h29m37.679794392s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:38.661400996 (13h29m38.250746043s) [upload=14002]: Done uploading segments... Aug 7 23:09:38.661423068 (13h29m38.250768135s) [upload=14002]: Upload finished. err: <nil> Aug 7 23:09:58.827549308 (13h29m58.416894295s) [upload=14009]: Starting upload Aug 7 23:09:58.827598442 (13h29m58.416943429s) [upload=14009]: Done starting upload Aug 7 23:09:58.827644020 (13h29m58.416989007s) [upload=14009]: Uploading segments... Aug 7 23:09:58.827674438 (13h29m58.417019425s) [upload=14009]: Got next segment. Inline: true Aug 7 23:09:58.827678135 (13h29m58.417023112s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:09:58.827711559 (13h29m58.417056546s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:58.827715526 (13h29m58.417060513s) [upload=14009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:58.827719804 (13h29m58.417064781s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:59.099262618 (13h29m58.688607605s) [upload=14009]: Done uploading segments... Aug 7 23:09:59.099279320 (13h29m58.688624297s) [upload=14009]: Upload finished. err: <nil> Aug 7 23:10:50.147486507 (13h30m49.736831494s) [upload=14023]: Starting upload Aug 7 23:10:50.147530131 (13h30m49.736875118s) [upload=14023]: Done starting upload Aug 7 23:10:50.147587851 (13h30m49.736932868s) [upload=14023]: Uploading segments... Aug 7 23:10:50.147650551 (13h30m49.736995568s) [upload=14023]: Got next segment. Inline: true Aug 7 23:10:50.147659338 (13h30m49.737004355s) [upload=14023]: Waiting for error group managing segments... Aug 7 23:10:50.147719804 (13h30m49.737064811s) [upload=14023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:50.147727708 (13h30m49.737072746s) [upload=14023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:50.147734752 (13h30m49.737079739s) [upload=14023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:50.421404977 (13h30m50.010749964s) [upload=14023]: Done uploading segments... Aug 7 23:10:50.421427519 (13h30m50.010772546s) [upload=14023]: Upload finished. err: <nil> Aug 7 23:11:30.587306004 (13h31m30.176650991s) [upload=14032]: Starting upload Aug 7 23:11:30.587348937 (13h31m30.176693924s) [upload=14032]: Done starting upload Aug 7 23:11:30.587398742 (13h31m30.17674373s) [upload=14032]: Uploading segments... Aug 7 23:11:30.587432447 (13h31m30.176777434s) [upload=14032]: Got next segment. Inline: true Aug 7 23:11:30.587437326 (13h31m30.176782313s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:11:30.587469467 (13h31m30.176814454s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:30.587472974 (13h31m30.176817951s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:30.587475168 (13h31m30.176820155s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:30.892529848 (13h31m30.481874835s) [upload=14032]: Done uploading segments... Aug 7 23:11:30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Duration: 1.845675455s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXXX.] duration: 6.206070379s FAIL (2025-08-07 23:16:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.20609188s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X...................X....................X...] duration: 6.407568062s FAIL (2025-08-07 23:16:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e85db788dbeec38ad8084daa355e1bf59b8b718af998fc40c7451ca1ba6627b2 != before 1cad814b0341363008cdb7d22dd66bb6d87f0105cd39850888ced4405a0b4b43 ======LOGS====== upload Aug 7 23:13:43.967401320 (13h33m43.556746307s) [upload=14069]: Uploading segments... Aug 7 23:13:43.967498206 (13h33m43.556843193s) [upload=14069]: Got next segment. Inline: false Aug 7 23:13:43.967560455 (13h33m43.556905442s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:43.967568550 (13h33m43.556913577s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:44.414536988 (13h33m44.003881975s) [seg_pos={0 0}, upload=14069]: Begin upload segment... Aug 7 23:13:44.414587104 (13h33m44.003932091s) [seg_pos={0 0}, upload=14069]: Done begin upload segment. Aug 7 23:13:44.414606933 (13h33m44.00395192s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414666236 (13h33m44.004011223s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414966160 (13h33m44.004311147s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415118381 (13h33m44.004463369s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415228682 (13h33m44.00457367s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415496494 (13h33m44.004841471s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415745370 (13h33m44.005090357s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416026788 (13h33m44.005371766s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416256569 (13h33m44.005601556s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416456080 (13h33m44.005801067s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416552746 (13h33m44.005897753s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416803596 (13h33m44.006148583s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417042221 (13h33m44.006387208s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417277101 (13h33m44.006622088s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417316085 (13h33m44.006661072s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417537459 (13h33m44.006882436s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417738975 (13h33m44.007083952s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417952763 (13h33m44.00729774s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:45.514309387 (13h33m45.103654374s) [upload=14069]: Next returned nil segment Aug 7 23:13:45.514327020 (13h33m45.103672007s) [upload=14069]: Waiting for error group managing segments... Aug 7 23:13:45.795258156 (13h33m45.384603143s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.802263469 (13h33m45.391608456s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.887877424 (13h33m45.477222401s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.350209185 (13h33m45.939554172s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.389672887 (13h33m45.979017874s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.399409194 (13h33m45.988754171s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.424971119 (13h33m46.014316106s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.450918913 (13h33m46.0402639s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.492609174 (13h33m46.081954161s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.516716376 (13h33m46.106061353s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.592315332 (13h33m46.181660319s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.746385819 (13h33m46.335730806s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.786539771 (13h33m46.375884758s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838001699 (13h33m46.427346676s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838017648 (13h33m46.427362636s) [seg_pos={0 0}, upload=14069]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:46.838217681 (13h33m46.427562658s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838422624 (13h33m46.427767601s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838575365 (13h33m46.427920352s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: rpc: dial tcp 104.36.82.16:4296: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838627195 (13h33m46.427972172s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:48060->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838635551 (13h33m46.427980538s) [seg_pos={0 0}, upload=14069]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:46.838702128 (13h33m46.428047115s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:46.838705895 (13h33m46.428050882s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.127771955 (13h33m46.717116932s) [upload=14069]: Done uploading segments... Aug 7 23:13:47.127791082 (13h33m46.717136069s) [upload=14069]: Upload finished. err: <nil> Duration: 6.407596788s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.908232602s FAIL (2025-08-07 23:16:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== segments... Aug 7 23:06:49.229716945 (13h26m48.819061932s) [upload=13950]: Upload finished. err: <nil> Aug 7 23:07:42.104963855 (13h27m41.694308842s) [upload=13968]: Starting upload Aug 7 23:07:42.105031205 (13h27m41.694376192s) [upload=13968]: Done starting upload Aug 7 23:07:42.105104063 (13h27m41.69444905s) [upload=13968]: Uploading segments... Aug 7 23:07:42.105139652 (13h27m41.694484639s) [upload=13968]: Got next segment. Inline: true Aug 7 23:07:42.105143259 (13h27m41.694488246s) [upload=13968]: Waiting for error group managing segments... Aug 7 23:07:42.105190129 (13h27m41.694535126s) [upload=13968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:42.105201120 (13h27m41.694546127s) [upload=13968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:42.105207281 (13h27m41.694552278s) [upload=13968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:42.417069033 (13h27m42.00641402s) [upload=13968]: Done uploading segments... Aug 7 23:07:42.417104130 (13h27m42.006449117s) [upload=13968]: Upload finished. err: <nil> Aug 7 23:08:34.230557646 (13h28m33.819902633s) [upload=13982]: Starting upload Aug 7 23:08:34.230604215 (13h28m33.819949193s) [upload=13982]: Done starting upload Aug 7 23:08:34.230655173 (13h28m33.82000016s) [upload=13982]: Uploading segments... Aug 7 23:08:34.230683928 (13h28m33.820028905s) [upload=13982]: Got next segment. Inline: true Aug 7 23:08:34.230687385 (13h28m33.820032362s) [upload=13982]: Waiting for error group managing segments... Aug 7 23:08:34.230719175 (13h28m33.820064162s) [upload=13982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:34.230722832 (13h28m33.820067819s) [upload=13982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:34.230725046 (13h28m33.820070033s) [upload=13982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:34.528494026 (13h28m34.117839013s) [upload=13982]: Done uploading segments... Aug 7 23:08:34.528512322 (13h28m34.117857309s) [upload=13982]: Upload finished. err: <nil> Aug 7 23:09:05.508570966 (13h29m5.097915953s) [upload=13989]: Starting upload Aug 7 23:09:05.508638426 (13h29m5.097983423s) [upload=13989]: Done starting upload Aug 7 23:09:05.508701747 (13h29m5.098046764s) [upload=13989]: Uploading segments... Aug 7 23:09:05.508743536 (13h29m5.098088533s) [upload=13989]: Got next segment. Inline: true Aug 7 23:09:05.508749488 (13h29m5.098094515s) [upload=13989]: Waiting for error group managing segments... Aug 7 23:09:05.508795345 (13h29m5.098140362s) [upload=13989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:05.508801848 (13h29m5.098146856s) [upload=13989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:05.508805736 (13h29m5.098150733s) [upload=13989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:05.806325549 (13h29m5.395670536s) [upload=13989]: Done uploading segments... Aug 7 23:09:05.806343342 (13h29m5.395688329s) [upload=13989]: Upload finished. err: <nil> Aug 7 23:09:38.090257337 (13h29m37.679602324s) [upload=14002]: Starting upload Aug 7 23:09:38.090300970 (13h29m37.679645957s) [upload=14002]: Done starting upload Aug 7 23:09:38.090372567 (13h29m37.679717554s) [upload=14002]: Uploading segments... Aug 7 23:09:38.090406682 (13h29m37.67975168s) [upload=14002]: Got next segment. Inline: true Aug 7 23:09:38.090410099 (13h29m37.679755076s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:09:38.090443282 (13h29m37.679788269s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:38.090447210 (13h29m37.679792198s) [upload=14002]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:38.090449415 (13h29m37.679794392s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:38.661400996 (13h29m38.250746043s) [upload=14002]: Done uploading segments... Aug 7 23:09:38.661423068 (13h29m38.250768135s) [upload=14002]: Upload finished. err: <nil> Aug 7 23:09:58.827549308 (13h29m58.416894295s) [upload=14009]: Starting upload Aug 7 23:09:58.827598442 (13h29m58.416943429s) [upload=14009]: Done starting upload Aug 7 23:09:58.827644020 (13h29m58.416989007s) [upload=14009]: Uploading segments... Aug 7 23:09:58.827674438 (13h29m58.417019425s) [upload=14009]: Got next segment. Inline: true Aug 7 23:09:58.827678135 (13h29m58.417023112s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:09:58.827711559 (13h29m58.417056546s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:58.827715526 (13h29m58.417060513s) [upload=14009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:58.827719804 (13h29m58.417064781s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:59.099262618 (13h29m58.688607605s) [upload=14009]: Done uploading segments... Aug 7 23:09:59.099279320 (13h29m58.688624297s) [upload=14009]: Upload finished. err: <nil> Aug 7 23:10:50.147486507 (13h30m49.736831494s) [upload=14023]: Starting upload Aug 7 23:10:50.147530131 (13h30m49.736875118s) [upload=14023]: Done starting upload Aug 7 23:10:50.147587851 (13h30m49.736932868s) [upload=14023]: Uploading segments... Aug 7 23:10:50.147650551 (13h30m49.736995568s) [upload=14023]: Got next segment. Inline: true Aug 7 23:10:50.147659338 (13h30m49.737004355s) [upload=14023]: Waiting for error group managing segments... Aug 7 23:10:50.147719804 (13h30m49.737064811s) [upload=14023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:50.147727708 (13h30m49.737072746s) [upload=14023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:50.147734752 (13h30m49.737079739s) [upload=14023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:50.421404977 (13h30m50.010749964s) [upload=14023]: Done uploading segments... Aug 7 23:10:50.421427519 (13h30m50.010772546s) [upload=14023]: Upload finished. err: <nil> Aug 7 23:11:30.587306004 (13h31m30.176650991s) [upload=14032]: Starting upload Aug 7 23:11:30.587348937 (13h31m30.176693924s) [upload=14032]: Done starting upload Aug 7 23:11:30.587398742 (13h31m30.17674373s) [upload=14032]: Uploading segments... Aug 7 23:11:30.587432447 (13h31m30.176777434s) [upload=14032]: Got next segment. Inline: true Aug 7 23:11:30.587437326 (13h31m30.176782313s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:11:30.587469467 (13h31m30.176814454s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:30.587472974 (13h31m30.176817951s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:30.587475168 (13h31m30.176820155s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:30.892529848 (13h31m30.481874835s) [upload=14032]: Done uploading segments... Aug 7 23:11:30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Duration: 1.90825315s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 1.900345734s FAIL (2025-08-07 23:16:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.900432138s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.011517378s FAIL (2025-08-07 23:16:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 46]: Upload finished. err: <nil> Aug 7 23:07:14.299823176 (13h27m13.889168163s) [upload=13956]: Starting upload Aug 7 23:07:14.299919721 (13h27m13.889264729s) [upload=13956]: Done starting upload Aug 7 23:07:14.300062905 (13h27m13.889407902s) [upload=13956]: Uploading segments... Aug 7 23:07:14.300136495 (13h27m13.889481482s) [upload=13956]: Got next segment. Inline: true Aug 7 23:07:14.300143029 (13h27m13.889488016s) [upload=13956]: Waiting for error group managing segments... Aug 7 23:07:14.300224825 (13h27m13.889569822s) [upload=13956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:14.300231578 (13h27m13.889576565s) [upload=13956]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:14.300235044 (13h27m13.889580041s) [upload=13956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:14.558772216 (13h27m14.148117193s) [upload=13956]: Done uploading segments... Aug 7 23:07:14.558788256 (13h27m14.148133243s) [upload=13956]: Upload finished. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.01154476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.000507827s FAIL (2025-08-07 23:16:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2.86683536s) [upload=13972]: Upload finished. err: <nil> Aug 7 23:09:06.935342508 (13h29m6.524687485s) [upload=13991]: Starting upload Aug 7 23:09:06.935397543 (13h29m6.52474252s) [upload=13991]: Done starting upload Aug 7 23:09:06.935446527 (13h29m6.524791514s) [upload=13991]: Uploading segments... Aug 7 23:09:06.935485321 (13h29m6.524830308s) [upload=13991]: Got next segment. Inline: true Aug 7 23:09:06.935489339 (13h29m6.524834326s) [upload=13991]: Waiting for error group managing segments... Aug 7 23:09:06.935521571 (13h29m6.524866558s) [upload=13991]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:06.935525147 (13h29m6.524870134s) [upload=13991]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:06.935527432 (13h29m6.524872419s) [upload=13991]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:07.220768192 (13h29m6.810113189s) [upload=13991]: Done uploading segments... Aug 7 23:09:07.220792449 (13h29m6.810137466s) [upload=13991]: Upload finished. err: <nil> Aug 7 23:09:35.163599610 (13h29m34.752944597s) [upload=14001]: Starting upload Aug 7 23:09:35.163659795 (13h29m34.753004782s) [upload=14001]: Done starting upload Aug 7 23:09:35.163720571 (13h29m34.753065558s) [upload=14001]: Uploading segments... Aug 7 23:09:35.163772821 (13h29m34.753117808s) [upload=14001]: Got next segment. Inline: true Aug 7 23:09:35.163777159 (13h29m34.753122146s) [upload=14001]: Waiting for error group managing segments... Aug 7 23:09:35.163816885 (13h29m34.753161872s) [upload=14001]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:35.163830291 (13h29m34.753175278s) [upload=14001]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:35.163832876 (13h29m34.753177863s) [upload=14001]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:35.429570925 (13h29m35.018915942s) [upload=14001]: Done uploading segments... Aug 7 23:09:35.429591264 (13h29m35.018936251s) [upload=14001]: Upload finished. err: <nil> Aug 7 23:09:54.175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000556529s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................................X] duration: 30.00071137s FAIL (2025-08-07 23:15:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000730425s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X.X.....................................................XXXX] duration: 1m0.001015657s FAIL (2025-08-07 23:15:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001036336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 30.000953451s FAIL (2025-08-07 23:15:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000978639s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 655.833748ms target: https://us1.storj.io/login FAIL (2025-08-07 23:15:33 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 655.856752ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001268554s FAIL (2025-08-07 23:15:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 23:07:17.101704242 (13h27m16.691049219s) [upload=13959]: Upload finished. err: <nil> Aug 7 23:07:52.992469001 (13h27m52.581813988s) [upload=13972]: Starting upload Aug 7 23:07:52.992519267 (13h27m52.581864254s) [upload=13972]: Done starting upload Aug 7 23:07:52.992562250 (13h27m52.581907237s) [upload=13972]: Uploading segments... Aug 7 23:07:52.992592087 (13h27m52.581937064s) [upload=13972]: Got next segment. Inline: true Aug 7 23:07:52.992595834 (13h27m52.581940821s) [upload=13972]: Waiting for error group managing segments... Aug 7 23:07:52.992631732 (13h27m52.581976719s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.992635890 (13h27m52.581980867s) [upload=13972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.992638755 (13h27m52.581983732s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.277462821 (13h27m52.866807808s) [upload=13972]: Done uploading segments... Aug 7 23:07:53.277490373 (13h27m52.86683536s) [upload=13972]: Upload finished. err: <nil> Aug 7 23:09:06.935342508 (13h29m6.524687485s) [upload=13991]: Starting upload Aug 7 23:09:06.935397543 (13h29m6.52474252s) [upload=13991]: Done starting upload Aug 7 23:09:06.935446527 (13h29m6.524791514s) [upload=13991]: Uploading segments... Aug 7 23:09:06.935485321 (13h29m6.524830308s) [upload=13991]: Got next segment. Inline: true Aug 7 23:09:06.935489339 (13h29m6.524834326s) [upload=13991]: Waiting for error group managing segments... Aug 7 23:09:06.935521571 (13h29m6.524866558s) [upload=13991]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:06.935525147 (13h29m6.524870134s) [upload=13991]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:06.935527432 (13h29m6.524872419s) [upload=13991]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:07.220768192 (13h29m6.810113189s) [upload=13991]: Done uploading segments... Aug 7 23:09:07.220792449 (13h29m6.810137466s) [upload=13991]: Upload finished. err: <nil> Aug 7 23:09:35.163599610 (13h29m34.752944597s) [upload=14001]: Starting upload Aug 7 23:09:35.163659795 (13h29m34.753004782s) [upload=14001]: Done starting upload Aug 7 23:09:35.163720571 (13h29m34.753065558s) [upload=14001]: Uploading segments... Aug 7 23:09:35.163772821 (13h29m34.753117808s) [upload=14001]: Got next segment. Inline: true Aug 7 23:09:35.163777159 (13h29m34.753122146s) [upload=14001]: Waiting for error group managing segments... Aug 7 23:09:35.163816885 (13h29m34.753161872s) [upload=14001]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:35.163830291 (13h29m34.753175278s) [upload=14001]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:35.163832876 (13h29m34.753177863s) [upload=14001]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:35.429570925 (13h29m35.018915942s) [upload=14001]: Done uploading segments... Aug 7 23:09:35.429591264 (13h29m35.018936251s) [upload=14001]: Upload finished. err: <nil> Aug 7 23:09:54.175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.00129231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X...................X....................X..] duration: 6.332730244s FAIL (2025-08-07 23:15:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e85db788dbeec38ad8084daa355e1bf59b8b718af998fc40c7451ca1ba6627b2 != before 1cad814b0341363008cdb7d22dd66bb6d87f0105cd39850888ced4405a0b4b43 ======LOGS====== upload Aug 7 23:13:43.967401320 (13h33m43.556746307s) [upload=14069]: Uploading segments... Aug 7 23:13:43.967498206 (13h33m43.556843193s) [upload=14069]: Got next segment. Inline: false Aug 7 23:13:43.967560455 (13h33m43.556905442s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:43.967568550 (13h33m43.556913577s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:44.414536988 (13h33m44.003881975s) [seg_pos={0 0}, upload=14069]: Begin upload segment... Aug 7 23:13:44.414587104 (13h33m44.003932091s) [seg_pos={0 0}, upload=14069]: Done begin upload segment. Aug 7 23:13:44.414606933 (13h33m44.00395192s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414666236 (13h33m44.004011223s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414966160 (13h33m44.004311147s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415118381 (13h33m44.004463369s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415228682 (13h33m44.00457367s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415496494 (13h33m44.004841471s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415745370 (13h33m44.005090357s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416026788 (13h33m44.005371766s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416256569 (13h33m44.005601556s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416456080 (13h33m44.005801067s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416552746 (13h33m44.005897753s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416803596 (13h33m44.006148583s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417042221 (13h33m44.006387208s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417277101 (13h33m44.006622088s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417316085 (13h33m44.006661072s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417537459 (13h33m44.006882436s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417738975 (13h33m44.007083952s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417952763 (13h33m44.00729774s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:45.514309387 (13h33m45.103654374s) [upload=14069]: Next returned nil segment Aug 7 23:13:45.514327020 (13h33m45.103672007s) [upload=14069]: Waiting for error group managing segments... Aug 7 23:13:45.795258156 (13h33m45.384603143s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.802263469 (13h33m45.391608456s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.887877424 (13h33m45.477222401s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.350209185 (13h33m45.939554172s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.389672887 (13h33m45.979017874s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.399409194 (13h33m45.988754171s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.424971119 (13h33m46.014316106s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.450918913 (13h33m46.0402639s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.492609174 (13h33m46.081954161s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.516716376 (13h33m46.106061353s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.592315332 (13h33m46.181660319s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.746385819 (13h33m46.335730806s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.786539771 (13h33m46.375884758s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838001699 (13h33m46.427346676s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838017648 (13h33m46.427362636s) [seg_pos={0 0}, upload=14069]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:46.838217681 (13h33m46.427562658s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838422624 (13h33m46.427767601s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838575365 (13h33m46.427920352s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: rpc: dial tcp 104.36.82.16:4296: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838627195 (13h33m46.427972172s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:48060->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838635551 (13h33m46.427980538s) [seg_pos={0 0}, upload=14069]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:46.838702128 (13h33m46.428047115s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:46.838705895 (13h33m46.428050882s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.127771955 (13h33m46.717116932s) [upload=14069]: Done uploading segments... Aug 7 23:13:47.127791082 (13h33m46.717136069s) [upload=14069]: Upload finished. err: <nil> Duration: 6.332756103s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.......................................................XXXX] duration: 30.000776845s FAIL (2025-08-07 23:15:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000797784s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.001171793s FAIL (2025-08-07 23:15:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01710319s) [upload=13938]: Upload finished. err: <nil> Aug 7 23:06:43.977164591 (13h26m43.566509578s) [upload=13946]: Starting upload Aug 7 23:06:43.977216861 (13h26m43.566561848s) [upload=13946]: Done starting upload Aug 7 23:06:43.977265654 (13h26m43.566610631s) [upload=13946]: Uploading segments... Aug 7 23:06:43.977299669 (13h26m43.566644646s) [upload=13946]: Got next segment. Inline: true Aug 7 23:06:43.977304398 (13h26m43.566649375s) [upload=13946]: Waiting for error group managing segments... Aug 7 23:06:43.977335879 (13h26m43.566680866s) [upload=13946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:06:43.977342241 (13h26m43.566687228s) [upload=13946]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:43.977344626 (13h26m43.566689613s) [upload=13946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:06:44.261001386 (13h26m43.850346383s) [upload=13946]: Done uploading segments... Aug 7 23:06:44.261036533 (13h26m43.85038152s) [upload=13946]: Upload finished. err: <nil> Aug 7 23:07:14.299823176 (13h27m13.889168163s) [upload=13956]: Starting upload Aug 7 23:07:14.299919721 (13h27m13.889264729s) [upload=13956]: Done starting upload Aug 7 23:07:14.300062905 (13h27m13.889407902s) [upload=13956]: Uploading segments... Aug 7 23:07:14.300136495 (13h27m13.889481482s) [upload=13956]: Got next segment. Inline: true Aug 7 23:07:14.300143029 (13h27m13.889488016s) [upload=13956]: Waiting for error group managing segments... Aug 7 23:07:14.300224825 (13h27m13.889569822s) [upload=13956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:14.300231578 (13h27m13.889576565s) [upload=13956]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:14.300235044 (13h27m13.889580041s) [upload=13956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:14.558772216 (13h27m14.148117193s) [upload=13956]: Done uploading segments... Aug 7 23:07:14.558788256 (13h27m14.148133243s) [upload=13956]: Upload finished. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001183757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.904645167s FAIL (2025-08-07 23:14:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== segments... Aug 7 23:06:49.229716945 (13h26m48.819061932s) [upload=13950]: Upload finished. err: <nil> Aug 7 23:07:42.104963855 (13h27m41.694308842s) [upload=13968]: Starting upload Aug 7 23:07:42.105031205 (13h27m41.694376192s) [upload=13968]: Done starting upload Aug 7 23:07:42.105104063 (13h27m41.69444905s) [upload=13968]: Uploading segments... Aug 7 23:07:42.105139652 (13h27m41.694484639s) [upload=13968]: Got next segment. Inline: true Aug 7 23:07:42.105143259 (13h27m41.694488246s) [upload=13968]: Waiting for error group managing segments... Aug 7 23:07:42.105190129 (13h27m41.694535126s) [upload=13968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:42.105201120 (13h27m41.694546127s) [upload=13968]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:42.105207281 (13h27m41.694552278s) [upload=13968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:42.417069033 (13h27m42.00641402s) [upload=13968]: Done uploading segments... Aug 7 23:07:42.417104130 (13h27m42.006449117s) [upload=13968]: Upload finished. err: <nil> Aug 7 23:08:34.230557646 (13h28m33.819902633s) [upload=13982]: Starting upload Aug 7 23:08:34.230604215 (13h28m33.819949193s) [upload=13982]: Done starting upload Aug 7 23:08:34.230655173 (13h28m33.82000016s) [upload=13982]: Uploading segments... Aug 7 23:08:34.230683928 (13h28m33.820028905s) [upload=13982]: Got next segment. Inline: true Aug 7 23:08:34.230687385 (13h28m33.820032362s) [upload=13982]: Waiting for error group managing segments... Aug 7 23:08:34.230719175 (13h28m33.820064162s) [upload=13982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:34.230722832 (13h28m33.820067819s) [upload=13982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:34.230725046 (13h28m33.820070033s) [upload=13982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:34.528494026 (13h28m34.117839013s) [upload=13982]: Done uploading segments... Aug 7 23:08:34.528512322 (13h28m34.117857309s) [upload=13982]: Upload finished. err: <nil> Aug 7 23:09:05.508570966 (13h29m5.097915953s) [upload=13989]: Starting upload Aug 7 23:09:05.508638426 (13h29m5.097983423s) [upload=13989]: Done starting upload Aug 7 23:09:05.508701747 (13h29m5.098046764s) [upload=13989]: Uploading segments... Aug 7 23:09:05.508743536 (13h29m5.098088533s) [upload=13989]: Got next segment. Inline: true Aug 7 23:09:05.508749488 (13h29m5.098094515s) [upload=13989]: Waiting for error group managing segments... Aug 7 23:09:05.508795345 (13h29m5.098140362s) [upload=13989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:05.508801848 (13h29m5.098146856s) [upload=13989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:05.508805736 (13h29m5.098150733s) [upload=13989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:05.806325549 (13h29m5.395670536s) [upload=13989]: Done uploading segments... Aug 7 23:09:05.806343342 (13h29m5.395688329s) [upload=13989]: Upload finished. err: <nil> Aug 7 23:09:38.090257337 (13h29m37.679602324s) [upload=14002]: Starting upload Aug 7 23:09:38.090300970 (13h29m37.679645957s) [upload=14002]: Done starting upload Aug 7 23:09:38.090372567 (13h29m37.679717554s) [upload=14002]: Uploading segments... Aug 7 23:09:38.090406682 (13h29m37.67975168s) [upload=14002]: Got next segment. Inline: true Aug 7 23:09:38.090410099 (13h29m37.679755076s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:09:38.090443282 (13h29m37.679788269s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:38.090447210 (13h29m37.679792198s) [upload=14002]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:38.090449415 (13h29m37.679794392s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:38.661400996 (13h29m38.250746043s) [upload=14002]: Done uploading segments... Aug 7 23:09:38.661423068 (13h29m38.250768135s) [upload=14002]: Upload finished. err: <nil> Aug 7 23:09:58.827549308 (13h29m58.416894295s) [upload=14009]: Starting upload Aug 7 23:09:58.827598442 (13h29m58.416943429s) [upload=14009]: Done starting upload Aug 7 23:09:58.827644020 (13h29m58.416989007s) [upload=14009]: Uploading segments... Aug 7 23:09:58.827674438 (13h29m58.417019425s) [upload=14009]: Got next segment. Inline: true Aug 7 23:09:58.827678135 (13h29m58.417023112s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:09:58.827711559 (13h29m58.417056546s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:58.827715526 (13h29m58.417060513s) [upload=14009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:58.827719804 (13h29m58.417064781s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:59.099262618 (13h29m58.688607605s) [upload=14009]: Done uploading segments... Aug 7 23:09:59.099279320 (13h29m58.688624297s) [upload=14009]: Upload finished. err: <nil> Aug 7 23:10:50.147486507 (13h30m49.736831494s) [upload=14023]: Starting upload Aug 7 23:10:50.147530131 (13h30m49.736875118s) [upload=14023]: Done starting upload Aug 7 23:10:50.147587851 (13h30m49.736932868s) [upload=14023]: Uploading segments... Aug 7 23:10:50.147650551 (13h30m49.736995568s) [upload=14023]: Got next segment. Inline: true Aug 7 23:10:50.147659338 (13h30m49.737004355s) [upload=14023]: Waiting for error group managing segments... Aug 7 23:10:50.147719804 (13h30m49.737064811s) [upload=14023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:50.147727708 (13h30m49.737072746s) [upload=14023]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:50.147734752 (13h30m49.737079739s) [upload=14023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:50.421404977 (13h30m50.010749964s) [upload=14023]: Done uploading segments... Aug 7 23:10:50.421427519 (13h30m50.010772546s) [upload=14023]: Upload finished. err: <nil> Aug 7 23:11:30.587306004 (13h31m30.176650991s) [upload=14032]: Starting upload Aug 7 23:11:30.587348937 (13h31m30.176693924s) [upload=14032]: Done starting upload Aug 7 23:11:30.587398742 (13h31m30.17674373s) [upload=14032]: Uploading segments... Aug 7 23:11:30.587432447 (13h31m30.176777434s) [upload=14032]: Got next segment. Inline: true Aug 7 23:11:30.587437326 (13h31m30.176782313s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:11:30.587469467 (13h31m30.176814454s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:30.587472974 (13h31m30.176817951s) [upload=14032]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:30.587475168 (13h31m30.176820155s) [upload=14032]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:30.892529848 (13h31m30.481874835s) [upload=14032]: Done uploading segments... Aug 7 23:11:30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Duration: 1.904674423s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [XX.X.....................................................XXX] duration: 1m0.000505737s FAIL (2025-08-07 23:14:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000524163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.338454729s FAIL (2025-08-07 23:14:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.338477734s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)